Hi dev team!
Update / suggestion. First of all, i think it is great that we have people working on our documentation. It is really important. But I can't help to note that the improvements, are going in an unmaintainable direction where we have to maintain the documentation at multiple locations. For instance our new toOneHot function already had some basic documentation inside its build-in script already [1], it would be much nicer if we modified that and then had some automation to generate the docs made in commit [2]. Now we have to maintain two locations, resulting in high risk of either being outdated. We need to ask ourselves where do we want to make the edit of documentation. I would argue that it should be located as close to the internal definition of a function, because then when making or modifying that function the documentation is also reviewed. To achieve this i suggest that we instead relocate the documentation source and generation to an appropriately named file inside [3], that simply loops through all function definitions from [4], and generate markdown files for our /docs/ folder. Shifting to such an implementation enables us to systematically cover all functions defined in the system. It also is a full list of the systems current build-in functions thereby giving us the exact file paths to extract the source documentation from the comments and generate the docs for those files. In the future it could be extended to verify and detecting which input and output types each function definition has, such that we have even less manual documentation needs. Best regards Sebastian [1] https://github.com/apache/systemml/blob/master/scripts/builtin/toOneHot.dml [2] https://github.com/apache/systemml/commit/859ad3a72906c67b7300c9980251da4cde9ed8f8 [3] https://github.com/apache/systemml/tree/master/src/main/java/org/apache/sysds/common [4] https://github.com/apache/systemml/blob/master/src/main/java/org/apache/sysds/common/Builtins.java ________________________________ From: arnab phani <phaniar...@gmail.com> Sent: Sunday, May 31, 2020 3:15:44 PM To: dev@systemml.apache.org Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. Hi, Another advantage of markdown syntax is that it gives more freedom to add a description as needed. Java methods and builtins might not fall in the same bucket and different builtins might need different ways to describe it. For example, glm needs more details than other built-ins. Too much standardization can be a bad idea. Regards, Arnab. On Sun, May 31, 2020 at 9:55 AM Baunsgaard, Sebastian <baunsga...@tugraz.at> wrote: > Hi Janardhan, > > > That would be one option, lets call it option 1, I have another suggestion: > > > 1. Markdown syntax with out commented individual lines. > 2. Jdoc like syntax with annotations > > Pros and cons > > * 1 Pros > * Easy Syntax most ppl know it. > * Easy to implement. > * 1 Cons > * Enables custom docs (everyone make their own distinct format > again) > * No standard way of verifying correctness of the docs (correct > parameter names etc.) > * No standard way of marking Input, return parameters. > * Requires modified Syntax with # at each line. where # normally > means header in markdown > * 2 Pros > * Easy Syntax Java ppl know it > * Standard way of making input > * Do not have to change any syntax at all from Jdoc > * Already supported syntax in DML > * 2 Cons > * Harder to implement since one would have to look into java doc > extraction and what is needed to support that, maybe we have to make such a > thing ourselves? > > I personally like option 2 with Jdoc more and then extending into > automatically parsing and making the either markdown files or HTML files > for the docs that you either way have to do in option 1. > > Best regards > Sebastian > ________________________________ > From: Janardhan <janardhan.pulivar...@gmail.com> > Sent: Sunday, May 31, 2020 9:25:43 AM > To: dev@systemml.apache.org > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > Hi Sebastian, :) > > Since the builtin DML files do not have full documentation yet, Can we > write > markdown syntax with the following heading in each dml file itself! > > 1. Function description > 2. Usage > 3. Arguments > 4. Returns > 5. Usage > > After that, as you've mentioned can be parsed with only removing # at the > start of each line, and write > all this data to one `builtins-reference.md` file. > > Thank you, > Janardhan > > On Tue, May 26, 2020 at 5:05 PM Baunsgaard, Sebastian < > baunsga...@tugraz.at> > wrote: > > > > > If this is meant as a question then, yes from me (this probably calls for > > a vote?), if: > > > > > > 1. someone is able to change the settings for the repository, to use > > the docs folder as the GitHub docs folder. > > 2. someone wants to make a PR containing the content of the gh-pages > > branch. > > > > Best regards > > Sebastian > > ________________________________ > > From: Janardhan <janardhan.pulivar...@gmail.com> > > Sent: Tuesday, May 26, 2020 1:22:11 PM > > To: dev@systemml.apache.org > > Subject: Re: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > Hi Sebastian, > > > > Can we move the `docs` folder in the existing `master` branch > > and merge the `gh-pages` branch to `docs` folder in the top-level > > directory. :) > > > > - Janardhan > > > > On Sun, May 24, 2020 at 3:28 PM Baunsgaard, Sebastian < > > baunsga...@tugraz.at> > > wrote: > > > > > Hi Janardhan, > > > > > > > > > Reworking the documentation seems like a really good idea! > > > > > > > > > A first step in my opinion is to start in the main repository. > > > > > > > > > I would suggest the following: > > > > > > > > > - Merge the current gh-pages branch into master docs folder. > > > > > > - Change the documentations page : http://apache.github.io/systemml/ SystemDS Documentation - SystemDS<http://apache.github.io/systemml/> apache.github.io SystemDS Documentation > to > > > use the docs folder for documentation. Should be doable inside the > > settings > > > on the repository. > > > > > > - Delete the gh-pages branch > > > > > > > > > I want this change because then new PRs can change the documentation in > > > one PR rather than two (which to be honest is not going to happen most > of > > > the time). > > > > > > The downside (we have to mention this) is that when you clone the > > > repository, you have slightly larger downloads, but i think the > trade-off > > > is fair. If you see any other issues please mention these in this > thread. > > > > > > > > > Next task I would like to suggest to include: > > > > > > > > > - Java docs generated and > > > > > > - Python docs generated to also be included as a sub-folder in docs, so > > > that the current webpage contains the docs for that to. > > > > > > > > > How to include these in a sensible manner is currently a good question, > > > and has to be seriously well considered. Including these will also > enable > > > us to move documentation closer to the individual code parts, making it > > > again more likely to get the documentation done. > > > > > > > > > Another more programmatic task is to make a parser for our DML files, > > such > > > that we can generate webpage documentation based on the documentation > > > syntax used (much like the python and java docs generator). This would > be > > > great since it will remove the duplication of documentation in the > > > individual DML files, and the documentation we have already in files > such > > > as: docs/dml-language-reference.md. > > > > > > This also gives us an excuse to clean up that documentation/scripts > which > > > has very different implementations in our scripts folder: > > > > > > > > > examples: > > > > > > > > > - <https://github.com/apache/systemml/tree/master/scripts> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/tree/master/scripts> systemml/scripts at master · apache/systemml · GitHub<https://github.com/apache/systemml/tree/master/scripts> github.com Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub. > > > > > > https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml> systemml/batch_norm1d.dml at master · apache/systemml · GitHub<https://github.com/apache/systemml/blob/master/scripts/nn/layers/batch_norm1d.dml> github.com Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub. > > > < > > > > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml> systemml/batch_norm2d.dml at master · Baunsgaard/systemml · GitHub<https://github.com/Baunsgaard/systemml/blob/master/scripts/nn/layers/batch_norm2d.dml> github.com Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development by creating an account on GitHub. > > > > > > > > > > - < > > > > > > https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml [https://avatars2.githubusercontent.com/u/9947148?s=400&v=4]<https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml> systemml/glm.dml at master · Baunsgaard/systemml · GitHub<https://github.com/Baunsgaard/systemml/blob/master/scripts/builtin/glm.dml> github.com Mirror of Apache SystemML. Contribute to Baunsgaard/systemml development by creating an account on GitHub. > > > > > > <https://github.com/apache/systemml/tree/master/scripts> [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/tree/master/scripts> systemml/scripts at master · apache/systemml · GitHub<https://github.com/apache/systemml/tree/master/scripts> github.com Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub. > > > > > > https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> systemml/Cox.dml at master · apache/systemml · GitHub<https://github.com/apache/systemml/blob/master/scripts/algorithms/Cox.dml> github.com Mirror of Apache SystemML. Contribute to apache/systemml development by creating an account on GitHub. > > > > > > > > > > > > Furthermore, I would suggest to postpone changing the main website: > > > > > > https://github.com/apache/systemml-website [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml-website> GitHub - apache/systemml-website: Mirror of Apache SystemML site<https://github.com/apache/systemml-website> github.com Mirror of Apache SystemML site. Contribute to apache/systemml-website development by creating an account on GitHub. > > > > > > But that task has to be started when the next release is scheduled, > > > because then we need to find out how to copy the current documentation > > to a > > > archive list of static webpages located: > > > > > > https://systemml.apache.org/documentation Documentation<https://systemml.apache.org/documentation> systemml.apache.org Project Documentation Page > > > > > > > > > best regards > > > > > > Sebastian > > > > > > ________________________________ > > > From: Janardhan <janard...@apache.org> > > > Sent: Sunday, May 24, 2020 11:02:53 AM > > > To: dev@systemml.apache.org > > > Subject: [DISCUSSION] Website stack `systemml.apache.org`. Thanks. > > > > > > Hi Sebastian, > > > > > > In our discussion[1][2] a while ago, there was a topic about changing > > > website > > > stack. > > > > > > Let us start a discussion about this in this thread. > > > > > > We are planning to work on this, after a feasibility study and tech > stack > > > vetting. > > > > > > Anybody would like to give suggestions would be great. > > > > > > [1] https://github.com/apache/systemml/pull/877 [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml/pull/877> [MINOR][DOC] Name refactor SystemML to SystemDS in Documentation by Baunsgaard · Pull Request #877 · apache/systemml · GitHub<https://github.com/apache/systemml/pull/877> github.com Brute change of documentation page to reflect name change. To fully enforce this two more steps are needed: https://github.com/apache/systemml-website Access to SystemML website server on https://systemml.apache.org/ Furthermore, we might want to consider starting fresh in the documentation. > > > [2] https://github.com/apache/systemml-website/pull/66 [https://avatars3.githubusercontent.com/u/47359?s=400&v=4]<https://github.com/apache/systemml-website/pull/66> [SYSTEMML-2539][BUILD] Upgrade build configuration by j143 · Pull Request #66 · apache/systemml-website · GitHub<https://github.com/apache/systemml-website/pull/66> github.com Upgrade gulp to 4.0 Since the task api is deprecated, fuctions were used, and exported as tasks. Documentation for gulpfile.js. Stick to Susy2, since mixins are not supported in 3.0 Upgrade npm dependencies to state-of-art. Commit package-lock.json file to version control. > > > > > > Thank you, > > > Janardhan > > > > > >
