Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Nicolas Goaziou writes: > So, indeed, the second failing test appears. It should pass now (at > least it does even on my compiled Org installation). I confirm the fix, thank you very much. > I think the others come from a file-error ("Opening output file"). I can see now why this happens: the temporary directory for the test should already exist — make usually takes care of that, but running it by hand does not. So these failures should go away if you "mkdir /tmp/tmp-orgtest" before starting the tests and you should also get a clean test run. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf rackAttack V1.04R1: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Achim Gratz writes: > Well, that is likely because you run the test on uncompiled source, > while I am running it on compiled orgmode... maybe one of these pesky > byte-compiler warnings shouldn't have been ignored. I can't reproduce > the 24 unexpected results, but they probably have a similar origin > (partially compiled orgmode?). It's possible. I have two different Org installations. One compiled for my work, and the other, uncompiled for development. Both are latest git version. When batch running tests on my compiled Org, I get: --8<---cut here---start->8--- Ran 130 tests, 105 results as expected, 25 unexpected (2012-03-04 15:42:25+0100) 9 expected failures 25 unexpected results: FAILED ob-emacs-lisp/commented-last-block-line-no-var FAILED ob-emacs-lisp/commented-last-block-line-with-var FAILED ob-exp/exports-both FAILED ob-exp/mixed-blocks-with-exports-both FAILED ob-exp/noweb-no-export-and-exports-both FAILED ob-exp/noweb-on-export FAILED ob-exp/noweb-on-export-with-exports-results FAILED ob-exp/noweb-strip-export-ensure-strips FAILED test-ob-lob/do-not-eval-lob-lines-in-example-blocks-on-export FAILED test-ob-sh/dont-error-on-empty-results FAILED test-ob/commented-last-block-line-no-var FAILED test-ob/commented-last-block-line-with-var FAILED test-ob/org-babel-remove-result--results-default FAILED test-org-babel/combining-scalar-and-raw-result-types FAILED test-org-babel/elisp-in-header-arguments FAILED test-org-babel/inline-src-blocks FAILED test-org-babel/inline-src_blk-default-results-replace-line-1 FAILED test-org-babel/just-one-results-block FAILED test-org-babel/multi-line-header-arguments FAILED test-org-babel/parse-header-args FAILED test-org-babel/parse-header-args2 FAILED test-org-babel/simple-named-code-block FAILED test-org-babel/simple-variable-resolution FAILED test-org-footnote/normalize-outside-org FAILED test-org-table/simple-formula --8<---cut here---end--->8--- So, indeed, the second failing test appears. It should pass now (at least it does even on my compiled Org installation). I think the others come from a file-error ("Opening output file"). Thanks for your help. Regards, -- Nicolas Goaziou
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Nicolas Goaziou writes: >> [...] >> 2 unexpected results: >>FAILED test-org-export/export-scope >>FAILED test-org-footnote/normalize-outside-org > > With your command, I was able to reproduce the first one (along with 24 > unexpected results), for which I have pushed a fix. I still don't see > the second one. Well, that is likely because you run the test on uncompiled source, while I am running it on compiled orgmode... maybe one of these pesky byte-compiler warnings shouldn't have been ignored. I can't reproduce the 24 unexpected results, but they probably have a similar origin (partially compiled orgmode?). Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Hello, Achim Gratz writes: > Nicolas Goaziou writes: >> I cannot reproduce any of them, interactively or in batch mode[1]. > > This is the invocation (to be run in bash, like make would do): > > TMPDIR=/tmp/tmp-orgtest /usr/local/bin/emacs -batch -Q \ > -L lisp/ -L testing/ -L contrib/lisp \ > --eval '(defconst org-release "7.8.03-Test")' -l testing/org-test.el \ > --eval '(require '"'"'org-element)' --eval '(require '"'"'org-export)' \ > --eval '(setq org-confirm-babel-evaluate nil)' -f org-test-run-batch-tests > > [...] > 2 unexpected results: >FAILED test-org-export/export-scope >FAILED test-org-footnote/normalize-outside-org With your command, I was able to reproduce the first one (along with 24 unexpected results), for which I have pushed a fix. I still don't see the second one. Regards, -- Nicolas Goaziou
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Nicolas Goaziou writes: > I cannot reproduce any of them, interactively or in batch mode[1]. This is the invocation (to be run in bash, like make would do): TMPDIR=/tmp/tmp-orgtest /usr/local/bin/emacs -batch -Q \ -L lisp/ -L testing/ -L contrib/lisp \ --eval '(defconst org-release "7.8.03-Test")' -l testing/org-test.el \ --eval '(require '"'"'org-element)' --eval '(require '"'"'org-export)' \ --eval '(setq org-confirm-babel-evaluate nil)' -f org-test-run-batch-tests [...] 2 unexpected results: FAILED test-org-export/export-scope FAILED test-org-footnote/normalize-outside-org > I'm not sure where they could come from. Since I run this with "-Q", I suspect that you implicitly rely on some customization someplace. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Wavetables for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldUserWavetables
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
"Sebastien Vauban" writes: > I don't understand why once there are 192 tests, once 130. I thought that the > second figure was the total number of tests, hence should be stable over the > test runs? The number of available tests depend on the Babel languages that are activated. One of those failures was with a test run that had all languages (except R) activated, the other one with no languages (except emacs-lisp, which is always active), but org-element and org-export made active. Those two also seem to pull in ob-sh and maybe other Babel languages, I would have to check. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ DIY Stuff: http://Synth.Stromeko.net/DIY.html
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Hello Achim, Achim Gratz wrote: > Two test failures in current HEAD: > > Test test-org-footnote/normalize-outside-org backtrace: > org-footnote-in-valid-context-p() > [...] > normal-top-level() > Test test-org-footnote/normalize-outside-org condition: > (void-variable message-cite-prefix-regexp) >FAILED 166/192 test-org-footnote/normalize-outside-org > > Test test-org-export/export-scope backtrace: > signal(ert-test-failed (((should (equal (org-export-as (quote test)) > [...] > normal-top-level() > Test test-org-export/export-scope condition: > (ert-test-failed > ((should >(equal > (org-export-as ...) > "text\n")) > :form > (equal "* Head1\n** Head2\ntext\n*** Head3\n" "text\n") > :value nil :explanation > (arrays-of-different-length 32 5 "* Head1\n** Head2\ntext\n*** Head3\n" > "text\n" first-mismatch-at 0))) >FAILED 95/130 test-org-export/export-scope I don't understand why once there are 192 tests, once 130. I thought that the second figure was the total number of tests, hence should be stable over the test runs? Best regards, Seb -- Sebastien Vauban
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Hello, Achim Gratz writes: > Two test failures in current HEAD: I cannot reproduce any of them, interactively or in batch mode[1]. I'm not sure where they could come from. Regards, [1] Though, I have 8 unexpected failures there, but not related to those described here. -- Nicolas Goaziou
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Two test failures in current HEAD: --8<---cut here---start->8--- Test test-org-footnote/normalize-outside-org backtrace: org-footnote-in-valid-context-p() org-footnote-at-reference-p() byte-code(\20\304\202 \305\n\306#\204\307\310\311\"\21\20 org-footnote-get-next-reference() org-footnote-normalize() (let ((major-mode (quote message-mode))) (org-footnote-normalize)) (progn (insert "Body[fn::def]\n-- \nFake signature\n-- \nSignature") (unwind-protect (progn (insert "Body[fn::def]\n-- \nFake signature\n (save-current-buffer (set-buffer temp-buffer) (unwind-protect (progn (with-current-buffer temp-buffer (unwind-protect (progn (insert "Bod (let ((temp-buffer (generate-new-buffer " *temp*"))) (with-current-b (with-temp-buffer (insert "Body[fn::def]\n-- \nFake signature\n-- \n (let ((org-footnote-tag-for-non-org-mode-files nil) (message-signatu (lambda nil (let ((org-footnote-tag-for-non-org-mode-files nil)) (wi byte-code("\306\307!▒q\210\310\216\311 \312\216\313\314\315\316\3 ert--run-test-internal([cl-struct-ert--test-execution-info [cl-struc byte-code("\306\307!\211▒r\310\311!q\210\312 d\313\223)L\210\314\216 ert-run-test([cl-struct-ert-test test-org-footnote/normalize-outside ert-run-or-rerun-test([cl-struct-ert--stats "\\(org\\|ob\\)" [[cl-st ert-run-tests("\\(org\\|ob\\)" #[(event-type &rest event-args) \30 ert-run-tests-batch("\\(org\\|ob\\)") ert-run-tests-batch-and-exit("\\(org\\|ob\\)") (let ((org-id-track-globally t) (org-id-locations-file (convert-stan org-test-run-batch-tests() call-interactively(org-test-run-batch-tests nil nil) command-execute(org-test-run-batch-tests) command-line-1(("-L" "lisp/" "-L" "testing/" "--eval" "(defconst org command-line() normal-top-level() Test test-org-footnote/normalize-outside-org condition: (void-variable message-cite-prefix-regexp) FAILED 166/192 test-org-footnote/normalize-outside-org --8<---cut here---end--->8--- --8<---cut here---start->8--- Test test-org-export/export-scope backtrace: signal(ert-test-failed (((should (equal (org-export-as (quote test)) ert-fail(((should (equal (org-export-as (quote test)) "text\n")) :fo (if (unwind-protect (setq value-2005 (apply fn-2003 args-2004)) (set (unless (unwind-protect (setq value-2005 (apply fn-2003 args-2004)) (let (form-description-2007) (unless (unwind-protect (setq value-200 (let ((value-2005 (quote ert-form-evaluation-aborted-2006))) (let (f (let ((fn-2003 (function equal)) (args-2004 (list (org-export-as (qu (should (equal (org-export-as (quote test)) "text\n")) (progn (fset (quote org-test-center-block) (function* (lambda (obj c (unwind-protect (progn (fset (quote org-test-center-block) (function (let* ((--cl-letf-bound-- (fboundp (quote org-test-center-block))) ( (letf (((symbol-function (quote org-test-center-block)) (function* ( (progn (fset (quote org-test-comment) (function* (lambda (obj conten (unwind-protect (progn (fset (quote org-test-comment) (function* (la (let* ((--cl-letf-bound-- (fboundp (quote org-test-comment))) (--cl- (letf (((symbol-function (quote org-test-comment)) (function* (lambd (progn (fset (quote org-test-comment-block) (function* (lambda (obj (unwind-protect (progn (fset (quote org-test-comment-block) (functio (let* ((--cl-letf-bound-- (fboundp (quote org-test-comment-block))) (letf (((symbol-function (quote org-test-comment-block)) (function* (progn (fset (quote org-test-drawer) (function* (lambda (obj content (unwind-protect (progn (fset (quote org-test-drawer) (function* (lam (let* ((--cl-letf-bound-- (fboundp (quote org-test-drawer))) (--cl-l (letf (((symbol-function (quote org-test-drawer)) (function* (lambda (progn (fset (quote org-test-dynamic-block) (function* (lambda (obj (unwind-protect (progn (fset (quote org-test-dynamic-block) (functio (let* ((--cl-letf-bound-- (fboundp (quote org-test-dynamic-block))) (letf (((symbol-function (quote org-test-dynamic-block)) (function* (progn (fset (quote org-test-example-block) (function* (lambda (obj (unwind-protect (progn (fset (quote org-test-example-block) (functio (let* ((--cl-letf-bound-- (fboundp (quote org-test-example-block))) (letf (((symbol-function (quote org-test-example-block)) (function* (progn (fset (quote org-test-export-block) (function* (lambda (obj c (unwind-protect (progn (fset (quote org-test-export-block) (function (let* ((--cl-letf-bound-- (fboundp (quote org-test-export-block))) ( (letf (((symbol-function (quote org-test-export-block)) (function* ( (progn (fset (quote org-test-fixed-width) (function* (lambda (obj co (unwind-protect (progn (fset (quote org-test-fixed-width) (function* (let* ((--cl-letf-bound-- (fboundp (quote org-test-fixed-width))) (- (letf (((symbol-function (quote org-test-fixed-width)) (function* (l (progn (fset (quote or
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Achim Gratz writes: > It all works now when adding "testing/" to the load path. I'll add > this to the default configuration in my Makefile fork. I've improved the test invocation in my Makefile fork. It now starts from a clean Emacs invocation withoutsite startup and loads the configured Babel languages and possibly extra packages before running the tests. All Babel languages are activated by default except R, since R seems to depend on ess (i.e. even if you add it to the list, the tests still fail due to that missing dependency). Extra packages, for instance from contrib, can be required by adding their names to BTEST_EXTRA in local.mk and possibly prepending or appending load-path elements or other command line options via BTEST_PRE and BTEST_POST. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Factory and User Sound Singles for Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Nicolas Goaziou writes: > This should be fixed now. Thank you for reporting it. I confirm the fix, testing is clean again, thank you very much. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptation for Waldorf microQ V2.22R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Hello, Achim Gratz writes: > There's another patch by Nicolas that added a few test, one of which > seems broken: This should be fixed now. Thank you for reporting it. Regards, -- Nicolas Goaziou
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Eric Schulte writes: [...] Thanks for your latest round of patches. It all works now when adding "testing/" to the load path. I'll add this to the default configuration in my Makefile fork. There's another patch by Nicolas that added a few test, one of which seems broken: OVERVIEW Mark set Test test-org-export/fuzzy-links backtrace: signal(error ("No link found")) error("No link found") org-open-at-point() (prog1 (goto-line 4) (org-open-at-point) (should (looking-at "<+ Factory and User Sound Singles for Waldorf rackAttack: http://Synth.Stromeko.net/Downloads.html#WaldorfSounds
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Eric Schulte writes: > My guess is that when I removed require statements for org-html and > org-ascii from from test-org-html and test-org-exp, they removed some > functionality not loaded by default in the test scripts. These should > probably be added into the Makefile. Unless someone can give me a hint, I'll have a look at this later... it'll probably become weekend before I get to this. > Yea, I also run most tests from my normal Emacs session, which is part > of the motivation for removing these embedded require statements. I'll see that I create a minimal config and use that just for testing from the Makefile (in my Makefile fork of course). If somebody has a good template to work from, please post, otherwise I should be able to scrape it together from the parts that have just been removed. BTW, where should that startup file go, directly under testing? Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ Samples for the Waldorf Blofeld: http://Synth.Stromeko.net/Downloads.html#BlofeldSamplesExtra
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Achim Gratz writes: > Eric Schulte writes: >> Moving forward on this point, many of the existing tests explicitly >> `require' new Org-mode functionality (mainly language support for >> testing code blocks execution). I do not think that tests should ever >> be activating new packages changing a users global environment. > > Thanks. > > I've had to deactivate the inclusion of ./testing/lisp/contrib into the > test flow due to a recent removal of said directory by Nicolas (he's > probably doing that patch himself) and now get this: > Yes, I talked with Nicolas this morning, and at my urging he moved everything from ./testing/lisp/contrib into ./testing/lisp and removed the contrib directory. I think moving forward we are best served by a single test directory, especially given that we already have good methods of conditionally loading only those tests appropriate for the users environment. > > lisp/org-mode> make test-dirty |& grep failed >failed5/109 ob-exp/export-from-a-temp-buffer >failed 17/109 org-missing-dependency/test-ob-C >failed 18/109 org-missing-dependency/test-ob-R >failed 19/109 org-missing-dependency/test-ob-awk >failed 20/109 org-missing-dependency/test-ob-fortran >failed 21/109 org-missing-dependency/test-ob-lilypond >failed 22/109 org-missing-dependency/test-ob-maxima >failed 23/109 org-missing-dependency/test-ob-octave >failed 24/109 org-missing-dependency/test-ob-python >failed 25/109 org-missing-dependency/test-org-element >failed 26/109 org-missing-dependency/test-org-export > > This is expected, as my local configuration doesn't have any of these > packages activated. > Yes, I assume that the lower case "failed" means expected failure. > > lisp/org-mode> make test-dirty | & grep FAILED >FAILED 27/109 test-ob-exp/org-babel-exp-src-blocks/w-no-file >FAILED 62/109 test-org-babel/get-src-block-info-body >FAILED 63/109 test-org-babel/get-src-block-info-language >FAILED 64/109 test-org-babel/get-src-block-info-tangle > > This is unexpected and I don't really see where they are coming from. > The backtrace always starts at (regexp-quote org-test-file-ob-anchor). > My guess is that when I removed require statements for org-html and org-ascii from from test-org-html and test-org-exp, they removed some functionality not loaded by default in the test scripts. These should probably be added into the Makefile. > >> For this reason I have just pushed up a commit which changes all >> >> (require 'org-foo) >> >> to >> >> (unless (featurep 'org-foo) >> (signal 'missing-test-dependency "Org support for doing foo.")) >> >> so that those tests simple aren't run on the users system. Please let >> me know if anyone thinks this is a mistake and we can discuss. The only >> drawback I see is that batch-mode scripts will have to explicitly >> activate the features which they would like to test, by evaluating forms >> like (require 'org-foo) before the call to the test suite -- and I would >> argue that being explicit about such things is a benefit. > > Well, currently all testing runs under my user settings, which is not > ideal for testing anyway. I suppose it would not be difficult to just > inject a different (minimal) startup file specifically for testing to > separate these issues. > Yea, I also run most tests from my normal Emacs session, which is part of the motivation for removing these embedded require statements. Cheers, > > > Regards, > Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Eric Schulte writes: > Moving forward on this point, many of the existing tests explicitly > `require' new Org-mode functionality (mainly language support for > testing code blocks execution). I do not think that tests should ever > be activating new packages changing a users global environment. Thanks. I've had to deactivate the inclusion of ./testing/lisp/contrib into the test flow due to a recent removal of said directory by Nicolas (he's probably doing that patch himself) and now get this: lisp/org-mode> make test-dirty |& grep failed failed5/109 ob-exp/export-from-a-temp-buffer failed 17/109 org-missing-dependency/test-ob-C failed 18/109 org-missing-dependency/test-ob-R failed 19/109 org-missing-dependency/test-ob-awk failed 20/109 org-missing-dependency/test-ob-fortran failed 21/109 org-missing-dependency/test-ob-lilypond failed 22/109 org-missing-dependency/test-ob-maxima failed 23/109 org-missing-dependency/test-ob-octave failed 24/109 org-missing-dependency/test-ob-python failed 25/109 org-missing-dependency/test-org-element failed 26/109 org-missing-dependency/test-org-export This is expected, as my local configuration doesn't have any of these packages activated. lisp/org-mode> make test-dirty | & grep FAILED FAILED 27/109 test-ob-exp/org-babel-exp-src-blocks/w-no-file FAILED 62/109 test-org-babel/get-src-block-info-body FAILED 63/109 test-org-babel/get-src-block-info-language FAILED 64/109 test-org-babel/get-src-block-info-tangle This is unexpected and I don't really see where they are coming from. The backtrace always starts at (regexp-quote org-test-file-ob-anchor). > For this reason I have just pushed up a commit which changes all > > (require 'org-foo) > > to > > (unless (featurep 'org-foo) > (signal 'missing-test-dependency "Org support for doing foo.")) > > so that those tests simple aren't run on the users system. Please let > me know if anyone thinks this is a mistake and we can discuss. The only > drawback I see is that batch-mode scripts will have to explicitly > activate the features which they would like to test, by evaluating forms > like (require 'org-foo) before the call to the test suite -- and I would > argue that being explicit about such things is a benefit. Well, currently all testing runs under my user settings, which is not ideal for testing anyway. I suppose it would not be difficult to just inject a different (minimal) startup file specifically for testing to separate these issues. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for KORG EX-800 and Poly-800MkII V0.9: http://Synth.Stromeko.net/Downloads.html#KorgSDada
Re: [O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Moving forward on this point, many of the existing tests explicitly `require' new Org-mode functionality (mainly language support for testing code blocks execution). I do not think that tests should ever be activating new packages changing a users global environment. For this reason I have just pushed up a commit which changes all (require 'org-foo) to (unless (featurep 'org-foo) (signal 'missing-test-dependency "Org support for doing foo.")) so that those tests simple aren't run on the users system. Please let me know if anyone thinks this is a mistake and we can discuss. The only drawback I see is that batch-mode scripts will have to explicitly activate the features which they would like to test, by evaluating forms like (require 'org-foo) before the call to the test suite -- and I would argue that being explicit about such things is a benefit. Cheers, Achim Gratz writes: > Tests for experimental org features (e.g. from contrib/ ) should expect > to fail when the user has not configured their inclusion into the > current setup. In other words, things like "(require org-element)" > should not break the test run, but instead just note that this test has > failed expectedly and continue testing. > > > Regards, > Achim. -- Eric Schulte http://cs.unm.edu/~eschulte/
[O] [Bug] Tests for experimental org-features should expect to fail if not activated by the user
Tests for experimental org features (e.g. from contrib/ ) should expect to fail when the user has not configured their inclusion into the current setup. In other words, things like "(require org-element)" should not break the test run, but instead just note that this test has failed expectedly and continue testing. Regards, Achim. -- +<[Q+ Matrix-12 WAVE#46+305 Neuron microQkb Andromeda XTk Blofeld]>+ SD adaptations for Waldorf Q V3.00R3 and Q+ V3.54R2: http://Synth.Stromeko.net/Downloads.html#WaldorfSDada