Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser
Sorry for delay. I expected that a next step with inline raw AST fragments would be easier, but I have got working example for parser tests grouped in single file earlier. Certainly org files may be source for parser tests. They are readable enough to be convenient for developers and, I hope, it would not require a lot of work to convert them in machine-readable files. However tests should be distributed in some format(s) with widely available parsers: s-expressions, YAML, JSON. I am against JSON for source files since working with multiline text will likely be a pain (besides editing of files, patches and VCS diffs should be taken into account). Actually there is a requirement for external projects related to parser already. Expected results are provided as S-expressions. For simplicity I described a test suite with a couple of cases using s-expressions (see attachments). Names of macro an functions are arbitrary. Interface should be like (demo-parse-test-suite "demo-test-suite.el") Current implementation creates an ert test for each entry in such file, but it may be changed later. I just had example of macro for parametrized test from earlier experiments. On 15/12/2021 20:23, Ihor Radchenko wrote: Max Nikulin writes: On 11/12/2021 21:39, Ihor Radchenko wrote: The attached patch is an attempt to create something like shared repo for Org element parser tests. "[PATCH]" prefix in the subject might be a reason why you message received less attention than it should. At least Tom and Timothy have their parsers, so I expected some response from them. I am unsure whether posts about Organice were from developers or from users. Patch is something too internal to development of namely Org Mode as Emacs major mode. Once I asked to avoid proposal to integrate everything meaning mostly particular person who was extremely active that time, but it had effect on more people than I thought. You intention may have stronger result than it should as well. I am not sure here. In a way, we already have such a format in test-org-element.el: (ert-deftest test-org-element/bold-parser () "Test `bold' parser." ;; Standard test. (should (org-test-with-temp-text "*bold*" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t It is a bit harder to parse than it is acceptable for external tools. However it may be converted to more convenient format using elisp facilities. So, in addition to writing the Org parser, third-party dev will also have to write a parser for the test format. I assume available parser for some wide spread format and some simple interpretation of the structures specific to these tests. Most existing parser libraries can handle individual files, but not individual pieces of text grouped into bigger file using yet another standard convention. It should take a several line of code to put string input into a temporary file. Likely it is possible to implement interface of file stream for input string. Finally, it may stimulate developers to add a method that parses strings. I found the following test corpus impressive and unmanageable in the form of separate files: https://github.com/tgbugs/laundry/blob/master/laundry/test.rkt But is there a standard multi-test in one file format that can be used in multiple programming languages? Some testing frameworks have facilities for parametrized tests with lists of arbitrary structures, others explicitly recommends to write a loop over such lists. Or even allow inline tests via buffer-local after-save-hook: - text :: "*Bold* emphasis" - keywords :: emphasis, heading - description :: Despite line is started from a star, there is no space after it, so it is not a heading. Upon save, the text:: field will be automatically converted to a test-hash.org+test-hash.el files. Or to a single file with some top-level metadata and a list (maybe nested lists) of tests. Unsure that such files should be stored in git. Version of Org and test set should be included into metadata for the whole suite. If the test set is distributed together with Org, I see no reason to do it. Otherwise, the test set should simply track bugfix and main branches of Org and follow Org releases. Does it sound reasonable? I have no particular opinion if test sources should be maintained in the same repository with Org or Emacs. If test are described in Org files then test suites should be distributed using some other way. Since partial compliance is assumed, format of test results should be declared as well to be able to publish overview or comparison. I provided a short description of the format in the README (see the original patch). Is it not enough? I mean result of *group* of tests to present comparison of parsers. Unsure if JUnit XML files are appropriate. Are properties like :begin and :end mandatory for reference results of parsing?
Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser
Max Nikulin writes: > On 11/12/2021 21:39, Ihor Radchenko wrote: >> >> The attached patch is an attempt to create something like shared repo >> for Org element parser tests. > > "[PATCH]" prefix in the subject might be a reason why you message > received less attention than it should. Well. I really wanted to set a technical tone of the discussion in contrast to my previous email (proposing this exact idea, among others) that generated little technical feedback and a lot of non-technical. > I think, the set should contain hundreds of tests to be helpful, thus 2 > files per test will likely be inconvenient since most of samples will be > short. I suggest to group test input and results into large files. Such > tests should be augmented by some metadata: keywords (labels, tags). > E.g. besides heading > > - sample: "* Simple Heading" >keywords: heading I am not sure here. In a way, we already have such a format in test-org-element.el: (ert-deftest test-org-element/bold-parser () "Test `bold' parser." ;; Standard test. (should (org-test-with-temp-text "*bold*" (org-element-map (org-element-parse-buffer) 'bold #'identity nil t The problem with grouping short tests into a single file is that we put an extra requirement on the testing code. The code will have to parse the test files with multiple tests, extract those tests, and then run the parser. So, in addition to writing the Org parser, third-party dev will also have to write a parser for the test format. I find individual files easier to get started with. Most existing parser libraries can handle individual files, but not individual pieces of text grouped into bigger file using yet another standard convention. The usual solution to the above problem is fixed test file format that can be processed by testing module of a given language. But is there a standard multi-test in one file format that can be used in multiple programming languages? Having said that, I like your idea about adding metadata to the tests. Probably, we can simply do it in the readme like the following: * Test group heading - file :: file:./test-files/test1.org - keywords :: k1, k2 - description :: description text Or even allow inline tests via buffer-local after-save-hook: - text :: "*Bold* emphasis" - keywords :: emphasis, heading - description :: Despite line is started from a star, there is no space after it, so it is not a heading. Upon save, the text:: field will be automatically converted to a test-hash.org+test-hash.el files. > Version of Org and test set should be included into metadata for the > whole suite. If the test set is distributed together with Org, I see no reason to do it. Otherwise, the test set should simply track bugfix and main branches of Org and follow Org releases. Does it sound reasonable? > Since partial compliance is assumed, format of test results should be > declared as well to be able to publish overview or comparison. I provided a short description of the format in the README (see the original patch). Is it not enough? > Are properties like :begin and :end mandatory for reference results of > parsing? They make structures more verbose and harder to read. Often it > is enough to compare content and similar properties. I afraid that if we put contents of every headline/element in place of :begin :end, the results will be even less readable. The results will have to dump multiple cumulative instances of the original file. Or do you have an alternative suggestion about the format of the reference parser output? >> Any comments or suggestions? >> I am particularly looking for thoughts about licensing and possible >> distribution of the test set in separate repository. > > Since these tests will unlikely become a part of some software, I do not > think that GPL may be an obstacle for any project. Requirement of signed > consent will likely prevent contributing of new cases from some developers. Actually, there have been talks about including Org mode tests into Emacs itself (https://yhetil.org/emacs-devel/87o8629h8g.fsf...@gmx.de/) Not all our tests are under GPL, but we may want to change this situation. Also, if we decide to not distribute this test set under GPL, what about the usual fear with someone contributing and its persons' employer coming on us with copyright claims? I do not have enough knowledge to judge this. Maybe Bastien or others know better. Or we may try to contact GNU legal team. Best, Ihor
Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser
Tim Cross writes: I have nothing to say about licence question. Replying to the other part > ... On > the other hand, I guess there could be cases where people want to just > checkout these samples to use to validate their own library/parser etc. > I tend towards putting them in the org-mode repository for simplicity. We can also advice to use sparse checkout as described in https://unix.stackexchange.com/questions/233327/is-it-possible-to-clone-only-part-of-a-git-project I just tried to do so for testing/ folder in Org repo. It was fairly fast. I guess that sparse clone should also be fine to prepare patches. Best, Ihor
Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser
Max Nikulin writes: > On 11/12/2021 21:39, Ihor Radchenko wrote: > > Since these tests will unlikely become a part of some software, I do not think > that GPL may be an obstacle for any project. Requirement of signed consent > will > likely prevent contributing of new cases from some developers. I agree. The test org files are input data rather than code and I don't think they fall under the copyright restrictions (or the code contribution guidelines). They probably don't even need to be GPL'd - possibly a CC license would be sufficient. As to whether they should be part of the org-mode repository or in their own repository, I'm not sure. It would be convenient to have them int he org-mode repository as I expect they will become part of the testing framework and only having to checkout one repository would be useful. On the other hand, I guess there could be cases where people want to just checkout these samples to use to validate their own library/parser etc. I tend towards putting them in the org-mode repository for simplicity.
Re: [PATCH] org-test: Create a collaborative test set for Org buffer parser
On 11/12/2021 21:39, Ihor Radchenko wrote: The attached patch is an attempt to create something like shared repo for Org element parser tests. "[PATCH]" prefix in the subject might be a reason why you message received less attention than it should. The test set is essentially a series of .org files alongside .el files containing normalised output of `org-element-parse-buffer'. (see the patch) I think, the set should contain hundreds of tests to be helpful, thus 2 files per test will likely be inconvenient since most of samples will be short. I suggest to group test input and results into large files. Such tests should be augmented by some metadata: keywords (labels, tags). E.g. besides heading - sample: "* Simple Heading" keywords: heading it should be possible to filter related cases with similar markup - sample: "*Bold* emphasis" keywords: emphasis, heading description: Despite line is started from a star, there is no space after it, so it is not a heading. - sample: " * Unordered list item" keywords: list, heading description: Due to leading space it is a list item, not a heading. - sample: "*" keywords: text, heading description: Not a heading since there is no space after the star. I omitted test IDs above. Version of Org and test set should be included into metadata for the whole suite. Since partial compliance is assumed, format of test results should be declared as well to be able to publish overview or comparison. Are properties like :begin and :end mandatory for reference results of parsing? They make structures more verbose and harder to read. Often it is enough to compare content and similar properties. Any comments or suggestions? I am particularly looking for thoughts about licensing and possible distribution of the test set in separate repository. Since these tests will unlikely become a part of some software, I do not think that GPL may be an obstacle for any project. Requirement of signed consent will likely prevent contributing of new cases from some developers.
[PATCH] org-test: Create a collaborative test set for Org buffer parser
Dear all,
The attached patch is an attempt to create something like shared repo
for Org element parser tests.
The idea is moving the tests out from Elisp into a set of text files.
That way, anyone interested in developing Org syntax support can use our
tests and potentially contribute more test files to the benefit of Org
mode for Emacs.
The test set is essentially a series of .org files alongside .el files
containing normalised output of `org-element-parse-buffer'. (see the
patch)
Anyone can contribute to the test set by adding new .org files and
generating the canonical parser output with new
M-x test-org-element-parser-save-expected-result function.
README.org in the repo also serves as a test file .
Any comments or suggestions?
I am particularly looking for thoughts about licensing and possible
distribution of the test set in separate repository.
Best,
Ihor
>From f7bb947517e8793a45864b614f06460d1132539d Mon Sep 17 00:00:00 2001
Message-Id:
From: Ihor Radchenko
Date: Sat, 11 Dec 2021 22:24:39 +0800
Subject: [PATCH] org-test: Create a separate testset for Org element parser
* testing/lisp/test-org-element-parser-sources/README.org: Add readme
file describing the test file format and organisation.
* testing/lisp/test-org-element-parser-sources/simple-heading.org: Add
an example test file.
* testing/lisp/test-org-element-parser-sources/README.el:
* testing/lisp/test-org-element-parser-sources/simple-heading.el: Add
normalised expected parser output files.
* testing/lisp/test-org-element-parser.el: New testset integration to
main Org test suite. The file defines ERT tests for files inside
test-org-element-parser-sources and an interactive function
`test-org-element-parser-save-expected-result' to generate parser
output files.
---
.../test-org-element-parser-sources/README.el | 42 ++
.../README.org| 64 +
.../simple-heading.el | 11 ++
.../simple-heading.org| 5 +
testing/lisp/test-org-element-parser.el | 129 ++
5 files changed, 251 insertions(+)
create mode 100644 testing/lisp/test-org-element-parser-sources/README.el
create mode 100644 testing/lisp/test-org-element-parser-sources/README.org
create mode 100644 testing/lisp/test-org-element-parser-sources/simple-heading.el
create mode 100644 testing/lisp/test-org-element-parser-sources/simple-heading.org
create mode 100644 testing/lisp/test-org-element-parser.el
diff --git a/testing/lisp/test-org-element-parser-sources/README.el b/testing/lisp/test-org-element-parser-sources/README.el
new file mode 100644
index 0..852df032f
--- /dev/null
+++ b/testing/lisp/test-org-element-parser-sources/README.el
@@ -0,0 +1,42 @@
+(org-data
+ (:begin 1 :contents-begin 2 :contents-end 1306 :end 1306 :post-affiliated 1 :post-blank 0)
+ (section
+ (:begin 2 :contents-begin 2 :contents-end 837 :end 838 :post-affiliated 2 :post-blank 1)
+ (paragraph
+ (:begin 2 :contents-begin 2 :contents-end 51 :end 52 :post-affiliated 2 :post-blank 1)
+ "This is a shared test suite for Org mode syntax.\n")
+ (paragraph
+ (:begin 52 :contents-begin 52 :contents-end 247 :end 248 :post-affiliated 52 :post-blank 1)
+ "The test suite consists of a number of .org example files alongside\nwith the expected parser output. Each .org file can be parsed as is\nand the result should match the corresponding .el file. \n")
+ (paragraph
+ (:begin 248 :contents-begin 248 :contents-end 424 :end 425 :post-affiliated 248 :post-blank 1)
+ "The parser results in .el files are Emacs sexps. Each sexp is an\noutput of "
+ (verbatim
+(:begin 324 :end 351 :post-blank 1))
+ "stripped from unessential\nproperties. Each sexp has the following form:\n")
+ (src-block
+ (:begin 425 :end 773 :post-affiliated 425 :post-blank 1))
+ (paragraph
+ (:begin 773 :contents-begin 773 :contents-end 837 :end 837 :post-affiliated 773 :post-blank 0)
+ "The properties of elements can be specified in arbitrary order.\n"))
+ (headline
+ (:archivedp nil :begin 838 :commentedp nil :contents-begin 854 :contents-end 1306 :end 1306 :footnote-section-p nil :level 1 :post-affiliated 838 :post-blank 0 :pre-blank 1 :priority nil :raw-value "Contributing" :tags nil :title
+ ("Contributing")
+ :todo-keyword nil :todo-type nil)
+ (section
+ (:begin 854 :contents-begin 854 :contents-end 1306 :end 1306 :post-affiliated 854 :post-blank 0)
+ (paragraph
+(:begin 854 :contents-begin 854 :contents-end 983 :end 984 :post-affiliated 854 :post-blank 1)
+"To add new test files to this suite, send a patch to Org mode mailing\nlist, as described in "
+(link
+ (:begin 946 :contents-begin nil :contents-end nil :end 981 :post-blank 0))
+".\n")
+ (paragraph
+(:begin 984 :contents-begin 984 :contents-end 1306 :end 1306 :post-affiliated 984 :post-blank 0)
+"The expected parser output can be generated using Emacs and latest\nversion of Org
