Hi Arnab, We have all the Java-based builtin functions documented at [1]. Most of these functions are already documented!
[1] http://apache.github.io/systemml/dml-language-reference.html#built-in-functions Thank you, Janardhan On Fri, Jun 5, 2020 at 4:37 PM arnab phani <phaniar...@gmail.com> wrote: > That's a good suggestion. One thing I want to point out that Builtins.java > [1] contains all the dml-bodied and java-bodied built-ins. For the dml > built-ins, it is easier to parse the corresponding dml script (if > documentation is available there), but for other built-ins, I am not aware > of any particular source today where they are documented -- a few of them > are documented as a part of commit message but not all. > Though it is certainly possible to slice the dml builtins from [1] out, but > it will be awesome to generate docs for all of those. > > [1] > > https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java > > Regards, > Arnab.. > > On Thu, Jun 4, 2020 at 12:44 PM Baunsgaard, Sebastian < > baunsga...@tugraz.at> > wrote: > > > Hi dev team! > > > > > > Update / suggestion. > > > > > > First of all, i think it is great that we have people working on our > > documentation. It is really important. But I can't help to note that the > > improvements, are going in an unmaintainable direction where we have to > > maintain the documentation at multiple locations. > > > > > > For instance our new toOneHot function already had some basic > > documentation inside its build-in script already [1], it would be much > > nicer if we modified that and then had some automation to generate the > docs > > made in commit [2]. Now we have to maintain two locations, resulting in > > high risk of either being outdated. > > > > > > We need to ask ourselves where do we want to make the edit of > > documentation. I would argue that it should be located as close to the > > internal definition of a function, because then when making or modifying > > that function the documentation is also reviewed. > > > > > > To achieve this i suggest that we instead relocate the documentation > > source and generation to an appropriately named file inside [3], that > > simply loops through all function definitions from [4], and generate > > markdown files for our /docs/ folder. > > > > > > Shifting to such an implementation enables us to systematically cover all > > functions defined in the system. It also is a full list of the systems > > current build-in functions thereby giving us the exact file paths to > > extract the source documentation from the comments and generate the docs > > for those files. > > > > In the future it could be extended to verify and detecting which input > and > > output types each function definition has, such that we have even less > > manual documentation needs. > > > > > > Best regards > > > > Sebastian > > > > > > [1] > > > https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml > > > > [2] > > > https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8 > > > > [3] > > > https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common > > > > [4] > > > https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java > > > > > > ________________________________ > > From: arnab phani <phaniar...@gmail.com> > > Sent: Sunday, May 31, 2020 3:15:44 PM > > To: dev@systemml.apache.org > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > Hi, > > > > Another advantage of markdown syntax is that it gives more freedom to > add a > > description as needed. Java methods and builtins might not fall in the > same > > bucket and different builtins might need different ways to describe it. > > For example, glm needs more details than other built-ins. Too much > > standardization can be a bad idea. > > > > Regards, > > Arnab. > > > > On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian < > > baunsga...@tugraz.at> > > wrote: > > > > > Hi Janardhan, > > > > > > > > > That would be one option, lets call it option 1, I have another > > suggestion: > > > > > > > > > 1. Markdown syntax with out commented individual lines. > > > 2. Jdoc like syntax with annotations > > > > > > Pros and cons > > > > > > * 1 Pros > > > * Easy Syntax most ppl know it. > > > * Easy to implement. > > > * 1 Cons > > > * Enables custom docs (everyone make their own distinct format > > > again) > > > * No standard way of verifying correctness of the docs (correct > > > parameter names etc.) > > > * No standard way of marking Input, return parameters. > > > * Requires modified Syntax with # at each line. where # normally > > > means header in markdown > > > * 2 Pros > > > * Easy Syntax Java ppl know it > > > * Standard way of making input > > > * Do not have to change any syntax at all from Jdoc > > > * Already supported syntax in DML > > > * 2 Cons > > > * Harder to implement since one would have to look into java doc > > > extraction and what is needed to support that, maybe we have to make > > such a > > > thing ourselves? > > > > > > I personally like option 2 with Jdoc more and then extending into > > > automatically parsing and making the either markdown files or HTML > files > > > for the docs that you either way have to do in option 1. > > > > > > Best regards > > > Sebastian > > > ________________________________ > > > From: Janardhan <janardhan.pulivar...@gmail.com> > > > Sent: Sunday, May 31, 2020 9:25:43 AM > > > To: dev@systemml.apache.org > > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > > > Hi Sebastian, :) > > > > > > Since the builtin DML files do not have full documentation yet, Can we > > > write > > > markdown syntax with the following heading in each dml file itself! > > > > > > 1. Function description > > > 2. Usage > > > 3. Arguments > > > 4. Returns > > > 5. Usage > > > > > > After that, as you've mentioned can be parsed with only removing # at > the > > > start of each line, and write > > > all this data to one `builtins-reference.md` file. > > > > > > Thank you, > > > Janardhan > > > > > > On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian < > > > baunsga...@tugraz.at> > > > wrote: > > > > > > > > > > > If this is meant as a question then, yes from me (this probably calls > > for > > > > a vote?), if: > > > > > > > > > > > > 1. someone is able to change the settings for the repository, to > use > > > > the docs folder as the GitHub docs folder. > > > > 2. someone wants to make a PR containing the content of the > gh-pages > > > > branch. > > > > > > > > Best regards > > > > Sebastian > > > > ________________________________ > > > > From: Janardhan <janardhan.pulivar...@gmail.com> > > > > Sent: Tuesday, May 26, 2020 1:22:11 PM > > > > To: dev@systemml.apache.org > > > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. > Thanks. > > > > > > > > Hi Sebastian, > > > > > > > > Can we move the `docs` folder in the existing `master` branch > > > > and merge the `gh-pages` branch to `docs` folder in the top-level > > > > directory. :) > > > > > > > > - Janardhan > > > > > > > > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian < > > > > baunsga...@tugraz.at> > > > > wrote: > > > > > > > > > Hi Janardhan, > > > > > > > > > > > > > > > Reworking the documentation seems like a really good idea! > > > > > > > > > > > > > > > A first step in my opinion is to start in the main repository. > > > > > > > > > > > > > > > I would suggest the following: > > > > > > > > > > > > > > > - Merge the current gh-pages branch into master docs folder. > > > > > > > > > > - Change the documentations page : > http://apache.github.io/systemml/ > > SystemDS Documentation - SystemDS<http://apache.github.io/systemml/> > > apache.github.io > > SystemDS Documentation > > > > > > > to > > > > > use the docs folder for documentation. Should be doable inside the > > > > settings > > > > > on the repository. > > > > > > > > > > - Delete the gh-pages branch > > > > > > > > > > > > > > > I want this change because then new PRs can change the > documentation > > in > > > > > one PR rather than two (which to be honest is not going to happen > > most > > > of > > > > > the time). > > > > > > > > > > The downside (we have to mention this) is that when you clone the > > > > > repository, you have slightly larger downloads, but i think the > > > trade-off > > > > > is fair. If you see any other issues please mention these in this > > > thread. > > > > > > > > > > > > > > > Next task I would like to suggest to include: > > > > > > > > > > > > > > > - Java docs generated and > > > > > > > > > > - Python docs generated to also be included as a sub-folder in > docs, > > so > > > > > that the current webpage contains the docs for that to. > > > > > > > > > > > > > > > How to include these in a sensible manner is currently a good > > question, > > > > > and has to be seriously well considered. Including these will also > > > enable > > > > > us to move documentation closer to the individual code parts, > making > > it > > > > > again more likely to get the documentation done. > > > > > > > > > > > > > > > Another more programmatic task is to make a parser for our DML > files, > > > > such > > > > > that we can generate webpage documentation based on the > documentation > > > > > syntax used (much like the python and java docs generator). This > > would > > > be > > > > > great since it will remove the duplication of documentation in the > > > > > individual DML files, and the documentation we have already in > files > > > such > > > > > as: docs/dml-language-reference.md. > > > > > > > > > > This also gives us an excuse to clean up that documentation/scripts > > > which > > > > > has very different implementations in our scripts folder: > > > > > > > > > > > > > > > examples: > > > > > > > > > > > > > > > - <https://github.com/apache/systemml/tree/master/scripts> > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > https://github.com/apache/systemml/tree/master/scripts> > > > > systemml/scripts at master · apache/systemml · GitHub< > > https://github.com/apache/systemml/tree/master/scripts> > > github.com > > Mirror of Apache SystemML. Contribute to apache/systemml development by > > creating an account on GitHub. > > > > > > > > > > > > > > > > > > > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml > > > > > > > systemml/batch_norm1d.dml at master · apache/systemml · GitHub< > > > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml > > > > > github.com > > Mirror of Apache SystemML. Contribute to apache/systemml development by > > creating an account on GitHub. > > > > > > > > > < > > > > > > > > > > > > > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml > > [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml > > > > > > > systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub< > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml > > > > > github.com > > Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development > > by creating an account on GitHub. > > > > > > > > > > > > > > > > > > > > - < > > > > > > > > > > > > > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml > > [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]< > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml > > > > > > > systemml/glm.dml at master · Baunsgaard/systemml · GitHub< > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml > > > > > github.com > > Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development > > by creating an account on GitHub. > > > > > > > > > > > > > > <https://github.com/apache/systemml/tree/master/scripts> > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > https://github.com/apache/systemml/tree/master/scripts> > > > > systemml/scripts at master · apache/systemml · GitHub< > > https://github.com/apache/systemml/tree/master/scripts> > > github.com > > Mirror of Apache SystemML. Contribute to apache/systemml development by > > creating an account on GitHub. > > > > > > > > > > > > > > > > > > > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> > > > > systemml/Cox.dml at master · apache/systemml · GitHub< > > > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> > > github.com > > Mirror of Apache SystemML. Contribute to apache/systemml development by > > creating an account on GitHub. > > > > > > > > > > > > > > > > > > > > > > > > Furthermore, I would suggest to postpone changing the main website: > > > > > > > > > > https://github.com/apache/systemml-website > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > https://github.com/apache/systemml-website> > > > > GitHub - apache/systemml-website: Mirror of Apache SystemML site< > > https://github.com/apache/systemml-website> > > github.com > > Mirror of Apache SystemML site. Contribute to apache/systemml-website > > development by creating an account on GitHub. > > > > > > > > > > > > > > But that task has to be started when the next release is scheduled, > > > > > because then we need to find out how to copy the current > > documentation > > > > to a > > > > > archive list of static webpages located: > > > > > > > > > > https://systemml.apache.org/documentation > > Documentation<https://systemml.apache.org/documentation> > > systemml.apache.org > > Project Documentation Page > > > > > > > > > > > > > > > > > > > best regards > > > > > > > > > > Sebastian > > > > > > > > > > ________________________________ > > > > > From: Janardhan <janard...@apache.org> > > > > > Sent: Sunday, May 24, 2020 11:02:53 AM > > > > > To: dev@systemml.apache.org > > > > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > > > > > > > Hi Sebastian, > > > > > > > > > > In our discussion[1][2] a while ago, there was a topic about > changing > > > > > website > > > > > stack. > > > > > > > > > > Let us start a discussion about this in this thread. > > > > > > > > > > We are planning to work on this, after a feasibility study and tech > > > stack > > > > > vetting. > > > > > > > > > > Anybody would like to give suggestions would be great. > > > > > > > > > > [1] https://github.com/apache/systemml/pull/877 > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > https://github.com/apache/systemml/pull/877> > > > > [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by > > Baunsgaard · Pull Request #877 · apache/systemml · GitHub< > > https://github.com/apache/systemml/pull/877> > > github.com > > Brute change of documentation page to reflect name change. To fully > > enforce this two more steps are needed: > > https://github.com/apache/systemml-website Access to SystemML website > > server on https://systemml.apache.org/ Furthermore, we might want to > > consider starting fresh in the documentation. > > > > > > > > > [2] https://github.com/apache/systemml-website/pull/66 > > [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]< > > https://github.com/apache/systemml-website/pull/66> > > > > [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request > > #66 · apache/systemml-website · GitHub< > > https://github.com/apache/systemml-website/pull/66> > > github.com > > Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used, > > and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, > since > > mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art. > > Commit package-lock.json file to version control. > > > > > > > > > > > > > > Thank you, > > > > > Janardhan > > > > > > > > > > > > > > >
