Re: [O] [bug] regression tests broken
Martyn Jago martyn.j...@btinternet.com writes: Hi Eric [...] Further to the problem of tests failing under sandboxed conditions unexpectedly, I've made a few observations which I want to pass on. * ID Currently the ID searching command `org-id-find' has global scope to find ID's all over the host system. Since developers frequently have 1 Orgs on their system, this can lead to confusion. For instance, the test `ob-C/table' in in test-ob-C.el appears to contain an orphan ID, which fails, but may well pass on the devs system, since the ID probably exists somewhere (does this test also require to be called test-ob-C/table)?. Unfortunately `org-id-find' seems to be heavily dependent on org-mode.el forms, and I suspect it would require a lot of time for me to attempt to restrict it's scope to `the current Org system in use, so I will leave it to someone with more experience. Another by-product is that sandboxed test systems are not guaranteed to pick up the _correct_ file containing the search ID (this plagued me this-morning). * `org-test-load' attempts to load symlinks (Emacs interlocking files). Currently `org-test-load' will happily attempt to load an interlocking symbolic link (such as .#test-ob-exp.el - martyn@88-96-171-138.59787). This results in a failure to run any tests, and is particularly annoying during test development. Below is a patch that fixes this problem for me. From 04dd7358295caef0d42a3fe93f4979ccd3b5c630 Mon Sep 17 00:00:00 2001 From: Martyn Jago martyn.j...@btinternet.com Date: Sun, 13 Nov 2011 14:43:34 + Subject: [PATCH] Avoid loading (and failing) symbolic links (interlocking files) * testing/org-test.el: During test development various interlocking files may be present in testing/lisp directory (since they are being edited by emacs). Currently org-test-load will attempt to load these and fail. --- testing/org-test.el |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/testing/org-test.el b/testing/org-test.el index 57b7252..ea8cae4 100644 --- a/testing/org-test.el +++ b/testing/org-test.el @@ -274,9 +274,10 @@ otherwise place the point at the beginning of the inserted text. (mapc (lambda (path) (if (file-directory-p path) - (rld path) + (rld path) (catch 'missing-test-dependency - (load-file path + (when (string-match ^[A-Za-z].*\\.el$ path) + load-file path (directory-files base 'full ^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$ (rld (expand-file-name lisp org-test-dir)) -- 1.7.3.4 I'm still looking at a potential variable passing problem during html export by inline calls, and will submit something as soon as I can. Best, Martyn
Re: [O] [bug] regression tests broken
Hi Eric Oops, the patch was broken - this one fixes the problem... [...] Currently `org-test-load' will happily attempt to load an interlocking symbolic link (such as .#test-ob-exp.el - martyn@88-96-171-138.59787). This results in a failure to run any tests, and is particularly annoying during test development. Below is a patch that fixes this problem for me. From b868e208dd3ece7c10818ed73f2cf0dc86dac94b Mon Sep 17 00:00:00 2001 From: Martyn Jago martyn.j...@btinternet.com Date: Sun, 13 Nov 2011 16:03:51 + Subject: [PATCH] Fix don't load symlinks (Emacs interlocking files) * testing/org-test.el: During test development various interlocking files may be present in testing/lisp directory (since they are being edited by emacs). Currently org-test-load will attempt to load these --- testing/org-test.el |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/testing/org-test.el b/testing/org-test.el index ea8cae4..7d2f7e7 100644 --- a/testing/org-test.el +++ b/testing/org-test.el @@ -276,8 +276,9 @@ otherwise place the point at the beginning of the inserted text. (if (file-directory-p path) (rld path) (catch 'missing-test-dependency - (when (string-match ^[A-Za-z].*\\.el$ path) - load-file path + (when (string-match ^[A-Za-z].*\\.el$ + (file-name-nondirectory path)) + (load-file path) (directory-files base 'full ^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$ (rld (expand-file-name lisp org-test-dir)) -- 1.7.3.4 Best, Martyn [...]
Re: [O] [bug] regression tests broken
Martyn Jago martyn.j...@btinternet.com writes: Hi Eric Eric Schulte schulte.e...@gmail.com writes: Hi Martyn, I have all tests passing in a branch in the Org-mode git repo (which I plan to merge into master soon). Could you try a run of this on the standard-code-block-syntax branch? I've added the standard-code-block-syntax branch to my test server, and have configured notification so that you will be informed via email on failure. Unfortunately, the branch is currently failing, as are all other builds including my test-branch. I will try to take a look tomorrow on why three tests are failing. I had to take special care that previous versions of Org-mode weren't loaded when running tests on this branch. The problem being that if some defvar forms are defined by a previous instillation of Org-mode their new values may not be overwritten with the latest versions in this branch. From looking at your test output it seems the only two tests which are failing are the most recent two related to nested code blocks. Again these are passing on my system so I'm not sure what the cause could be. As a side note, how automated is your crusecontrol.rb setup? Would you recommend this as a way to run automated tests on the Org-mode server after every git commit? It is completely autonomous, git-pulling, cleaning, building, building docs, and running tests within 10 minutes or so of a fresh commit to the Org repository. Builds are serialized. I would certainly recommend it as an easy way of setting up a test server, since everything you need is built-in, such as email notification (notifying the list is of course a possibility), RSS, and rapid backtrace investigation etc. I have come across other gnu projects that use cruisecontrolrb. Your server would need Ruby installed (I use v1.9.2), but installation should be straight-forward since I have already written the scripts, which is fairly simple anyway. Hope that helps This sounds like a very nice setup. It would be great if you can work with me and Jason to see if we can set this up on the Org-mode server. Thanks -- Eric Best Martyn Thanks -- Eric Martyn Jago martyn.j...@btinternet.com writes: Hi All automated regression tests broken since 7720786 commit http://martynjago.com: Best, Martyn -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] [bug] regression tests broken
Martyn Jago martyn.j...@btinternet.com writes: Martyn Jago martyn.j...@btinternet.com writes: Hi Eric [...] Further to the problem of tests failing under sandboxed conditions unexpectedly, I've made a few observations which I want to pass on. * ID Currently the ID searching command `org-id-find' has global scope to find ID's all over the host system. Since developers frequently have 1 Orgs on their system, this can lead to confusion. Customizing the `org-id-locations-file' can be used to isolate ID lists between different instances of Emacs. Only the files in the `org-id-files' list should be scanned for IDs, so it shouldn't be too difficult to limit the scope of those files used in testing to only the example files in the testing directory. For instance, the test `ob-C/table' in in test-ob-C.el appears to contain an orphan ID, which fails, Confirmed, I don't have this ID located on my system either. Perhaps the author of this test file can help here. Note that this test is marked as expected to fail, so it should not affect the test suite as a whole. but may well pass on the devs system, since the ID probably exists somewhere (does this test also require to be called test-ob-C/table)?. No, the existing name is fine, see the ERT test match string in `org-test-run-all-tests' to see which tests will be picked up by the Org-mode test suite.. Unfortunately `org-id-find' seems to be heavily dependent on org-mode.el forms, and I suspect it would require a lot of time for me to attempt to restrict it's scope to `the current Org system in use, so I will leave it to someone with more experience. Another by-product is that sandboxed test systems are not guaranteed to pick up the _correct_ file containing the search ID (this plagued me this-morning). Given that all of these IDs are long hash tags I doubt there is any risk of ID duplication unless there are multiple checkouts of Org-mode being included in the files Org-mode scans for IDs. * `org-test-load' attempts to load symlinks (Emacs interlocking files). Currently `org-test-load' will happily attempt to load an interlocking symbolic link (such as .#test-ob-exp.el - martyn@88-96-171-138.59787). This results in a failure to run any tests, and is particularly annoying during test development. Below is a patch that fixes this problem for me. Great, I've just applied this patch. Thanks -- Eric From 04dd7358295caef0d42a3fe93f4979ccd3b5c630 Mon Sep 17 00:00:00 2001 From: Martyn Jago martyn.j...@btinternet.com Date: Sun, 13 Nov 2011 14:43:34 + Subject: [PATCH] Avoid loading (and failing) symbolic links (interlocking files) * testing/org-test.el: During test development various interlocking files may be present in testing/lisp directory (since they are being edited by emacs). Currently org-test-load will attempt to load these and fail. --- testing/org-test.el |5 +++-- 1 files changed, 3 insertions(+), 2 deletions(-) diff --git a/testing/org-test.el b/testing/org-test.el index 57b7252..ea8cae4 100644 --- a/testing/org-test.el +++ b/testing/org-test.el @@ -274,9 +274,10 @@ otherwise place the point at the beginning of the inserted text. (mapc (lambda (path) (if (file-directory-p path) -(rld path) + (rld path) (catch 'missing-test-dependency - (load-file path + (when (string-match ^[A-Za-z].*\\.el$ path) +load-file path (directory-files base 'full ^\\([^.]\\|\\.\\([^.]\\|\\..\\)\\).*\\.el$ (rld (expand-file-name lisp org-test-dir)) -- Eric Schulte http://cs.unm.edu/~eschulte/
Re: [O] [bug] regression tests broken
Hi Eric Eric Schulte schulte.e...@gmail.com writes: [...] This sounds like a very nice setup. It would be great if you can work with me and Jason to see if we can set this up on the Org-mode server. No problem. If it is a Linux server, I can work on setting up a fresh instance on Linux prior the actual deployment (currently on OSX). Best, Martyn Thanks -- Eric Best Martyn Thanks -- Eric Martyn Jago martyn.j...@btinternet.com writes: Hi All automated regression tests broken since 7720786 commit http://martynjago.com: Best, Martyn
Re: [O] [bug] regression tests broken
Hi Eric Eric Schulte schulte.e...@gmail.com writes: Hi Martyn, I have all tests passing in a branch in the Org-mode git repo (which I plan to merge into master soon). Could you try a run of this on the standard-code-block-syntax branch? I've added the standard-code-block-syntax branch to my test server, and have configured notification so that you will be informed via email on failure. Unfortunately, the branch is currently failing, as are all other builds including my test-branch. I will try to take a look tomorrow on why three tests are failing. As a side note, how automated is your crusecontrol.rb setup? Would you recommend this as a way to run automated tests on the Org-mode server after every git commit? It is completely autonomous, git-pulling, cleaning, building, building docs, and running tests within 10 minutes or so of a fresh commit to the Org repository. Builds are serialized. I would certainly recommend it as an easy way of setting up a test server, since everything you need is built-in, such as email notification (notifying the list is of course a possibility), RSS, and rapid backtrace investigation etc. I have come across other gnu projects that use cruisecontrolrb. Your server would need Ruby installed (I use v1.9.2), but installation should be straight-forward since I have already written the scripts, which is fairly simple anyway. Hope that helps Best Martyn Thanks -- Eric Martyn Jago martyn.j...@btinternet.com writes: Hi All automated regression tests broken since 7720786 commit http://martynjago.com: Best, Martyn
[O] [bug] regression tests broken
Hi All automated regression tests broken since 7720786 commit http://martynjago.com: Best, Martyn
Re: [O] [bug] regression tests broken
Hi Martyn, I have all tests passing in a branch in the Org-mode git repo (which I plan to merge into master soon). Could you try a run of this on the standard-code-block-syntax branch? As a side note, how automated is your crusecontrol.rb setup? Would you recommend this as a way to run automated tests on the Org-mode server after every git commit? Thanks -- Eric Martyn Jago martyn.j...@btinternet.com writes: Hi All automated regression tests broken since 7720786 commit http://martynjago.com: Best, Martyn -- Eric Schulte http://cs.unm.edu/~eschulte/
