Re: [Orgmode] Re: How you can help
Avdi Grimm [EMAIL PROTECTED] writes: On Thu, Oct 23, 2008 at 11:49 AM, Richard Riley [EMAIL PROTECTED] wrote: The nature of OSS means that the community using the product keep it stable and bug free. I dont think the efforts to produce meaningful regression tests would be beneficial in an ever morphing product like org-mode. Clearly my humble opinion on that one :-; I don't know of a single open-source project with more than one or two developers which doesn't have a suite of regression tests. In fact, I know of a number of OSS projects which *will not* accept a patch unless tests accompany it. Once a codebase grows beyond a one-person pet project, tests become increasingly necessary for maintainability. Then I have to disagree. There are many OSS projects with no regression tests. Is this a good thing? Not necessarily. But the nature of emacs, elisp and the customization features make it, for me, a slightly different kettle of fish. In fact the nature of this thread and the nature of most emacs customizations suggests to me that the regression tests are merely using the SW and seeing if it works as expected - not an automated process. This is the important piece : automated regression tests. Regression testing as in run it and see if it works is fine - automated ones are, well, another issue. Lets see how this pans out. Be sure I am not against testing, regression testing etc, I am merely suggesting that automated tests are possibly less effective than a proper beta release to users which is currently provided. -- Do you realize if it weren't for Edison we'd be watching TV by candlelight? ~Al Boliska ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
Hi Richard,
Richard Riley wrote:
Sebastian Rose [EMAIL PROTECTED] writes:
Hi Richard,
Richard Riley [EMAIL PROTECTED] writes:
Added this one to the Clippboard section on new org-tests/index.org in
Worg.git. (this section will be temporary...)
Something like the above should only be a link (at most) to the emacs
manual. Reproducing standard info is bad in the long run in case things
in the base product (emacs) change for example.
On the long run, yes. I just added this section, to start the page. And
since it's just STARTing... this was in the first or second reaction
that came in. Feel free to edit ('go wild' ;-))!
I wouldn't bother to have several pages like this one will be (it's
supposed to be an index on the long run), each covering another way of
testing.
I just meant to take _ACT_. Scepticism is a good thing, as long as it
doesn't stop you from doing.
Of course not.
Glad you replied. While I speek a little English, I'm not alwas shure
how it _feels_ for native speakers. For example this one about
scepticism.
Some kind of regression testing framework would be awesome. Org-mode is
large enough that this is almost a necessity to keep things stable and
bug-free.
It's big and feature-RICH.
The nature of OSS means that the community using the product keep it
stable and bug free. I dont think the efforts to produce meaningful
regression tests would be beneficial in an ever morphing product like
org-mode. Clearly my humble opinion on that one :-;
We do test our software by using it. But the bug in the HTML-exporter
that Carsten has fixed two days ago, was introduced in early September
and would be in 6.10, which is supposed to be in the emacs 23 release.
A very simple test plan would have revealed it.
The question is : define simple. I am not against test plans, but the
nature of such means a process in place where such test plans are
carried out. Carsten knows his SW inside out and would find it hard to
apply test cases every time a bug is fixed for time alone. Bugs happen
in SW and the nature (excuse the repetition) of OSS is that we, the
users, are kind of the testers and users.
OK, simplicity.
That's an important point I'd like to make clear. I _want_ simplicity
in two ways:
a) This here is _not_ meant to make Carsten work _more_, but to _help_
Carsten and all contributors. We must keep that in mind, while
searching for the right way to test.
Not Carsten, but _we_ collect the bugs, _we_ write the
tests (whithout interfering with Carstens code), _we_ setup the
testenvironment, and _we_ finally _do_ the testing.
b) I _want_ tests, that everyone can do. Nobody knows, who of us is
still here in a year or two. I like the idea of automated testing,
but I'd also like to get every org-user to test. Imagine to
have org-test.el in contrib (split into modules maybe).
(setq org/test-html-export-base-dir ~/org-test/data/)
(setq org/test-html-export-publising-dir ~/public_html/org-test/)
M-x org/test-test-html-export
The whole thing automatically executed once a week/day/mounth,
after every git-pull, SCHEDULED +1d, sending a backtrace on
failure.
Once tests exist, we will find out how the interaction with Carsten
works best / where neccessary. I know it will be hard to adjust all
those tests to 'movin target' called Org.
Maybe something can be put together from the git testing framework and
use of emacs -batch to process test org files and verify the output is
as expected (with diff or some other tool).
Hey, diff is a good idea!!
I didn't take the verification of the output into account yet :-)
I just pushed a change of Worgs start page, and added a directory
'org-tests'. I've placed an index.org there, which now is just a
collection of ideas (I'm on my day job, so I can't really work on it
now).
Don't know how often the git repo is published.
Bernt and Ben, are you 'worgers' allready?
Do you think it makes sense to add snippets and ideas to the new page in
Worg? I think while the list great to exchange ideas, it's good to have
a place, where all those ideas are destilled to one-liners.
I must say I am dubious about this. It means, for the tests to be
meaningful, that the output must be a fixed format in base org.
Why? If the test bails out 'ERROR', I will have to look for the
reason. If the format changed in a legal way = adjust the test.
What is an error? And for whom? That the categories are one more tab to
the right?
In this case:
if I have a project defined in org-publish-projects-alist,
and :base-dir contains a file /foo/bar.org,
and, when publishing,
this file goes to /bar.html (instead of /foo/bar.html),
it is definitively an ERROR. At least in my eyes :-)
I know I am probably sounding a bit of a killjoy here but
having been involved in closed source SW development with dedicated QA
and testing teams it was hard enough then. Maybe I am just being a bit
of an old curmudgeon
Re: [Orgmode] Re: How you can help
On Fri, Oct 24, 2008 at 5:43 PM, Richard Riley wrote: [snip] I think the best thing people can do is keep a stable version and also test the CVS (oops) GIT version too very now and again. +1 That's exactly what I try to do. I update almost every other day, go through (using tig) the diffs of code, changelog, documentation, comments.. and see if I can try out and learn something. I also try and report if something doesn't work right for me with whatever details I can provide. It really doesn't take too long. I definitely need to learn more about generating backtraces and debuggers so I could do a better job at providing right kind and level of details. With the kind of people we have on the list, we already have a high quallity Volunteer Powered Massively Parallel Testing System - VPMPTS (tm) in place. :) Let me take a step back and think aloud why we need a bug-tracking and testing system (if only to clarify my understanding) and who/when does it help. Following scenarios come to mind (and how they can be handled best (again only my limited understanding)): 1. Someone sees something odd, assumes it's a bug and wants/needs to report it. Report it to list, people will help out if it's a configuration or understanding issue or confirm if it's a repeatble issue. 2. A bug report needs to be confirmed. VPMPTS goes to work. :) 3. A bug needs to be communicated to the developers. Developers look at the bug confirmation reports on the list and decide whether to accept it as a bug or improve our collective understanding. 4. New functionality needs to be tested for regression. VPMPTS to rescue! 5. Bug reporter needs to know when the bug is fixed. Read the list! (Possibly track git repo as well.) 6. Developers needs to sync up about who is working on a specific bug. Seems like bug tracker will be useful. This is for developers to comment upon but bug acknowledging developer can be safely assumed to know the reasons behind the bug and assumed to own it. I confess I do not have programming or testing background so I may be completely out of whack here. IMHO, we should go in the direction of taking on the overhead of maintaining additional infrastructure only when it's inevitable or really valuable. Let me take another step back and recall Carsten't first email on this thread; he asked for help in 5 specific areas: 1. Providing sufficient details when reporting bugs. 2. Reproducing reported bugs. 3. Sub-system adoption. 4. Answering emails on list. 5. Adding to Worg in form of FAQ, tutorials, and hacks. I think #1 needs a tutorial (at least for me.) We already do #2 and #4. #5 needs more work to cover as many features as possible (mea culpa.) #3 is up to elisp gurus here. I believe Carsten thought long and hard before he came up with this list of very specific areas which, IMHO, covers most (if not all) of the scenarios I conjured up above. Do I think regression testing is important? Yes - in certain environments. But every time Carsten, you, myself or anyone else fires up org-mode we are already doing just that. Yes, but we can do better, easier and more complete. Possibly. I look forward to seeing proposed solutions and would gladly lend some time to applying them. Ditto. Please do not assume that I am against any change in our systems and processes. I am just trying to see the system working in my mind's eye. -- Manish ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
On Fri, Oct 24, 2008 at 12:27 PM, Manish [EMAIL PROTECTED] wrote: Let me take a step back and think aloud why we need a bug-tracking and testing system (if only to clarify my understanding) and who/when does it help. Following scenarios come to mind (and how they can be handled best (again only my limited understanding)): First of all, bug tracking are completely separate beasts, and should be considered separately. You can have either one without the other. Second: I don't care as much about bug tracking, except for the following case: if there is a bug tracker and I'm feeling bored and want to contribute a little, I'll check it for something that I think I can fix. If there is no such list, I won't do this, and I'll contribute less. I do *not* follow the mailing list closely enough to pick up bugs from there except by sheer chance, and this is not going to change. I really don't care where or how the list is kept, so long as there is a list. Third, regarding testing. I'm a coder, I know some ELisp, I *love* org-mode, and I would like to contribute. That said, I am completely uninterested in doing manual QA for Org. I am not going to go clicking around just to look for broken behavior. I have neither the time nor the interest. However, if there is an automated test suite I can promise you that a) I will run it before submitting any change I make; and b) I will add new tests to cover any functionality I add. However, you should take all of this with a grain of salt because I haven't submitted any code at all yet. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
On 2008-Oct-23, at 16:49, Richard Riley wrote: Sebastian Rose [EMAIL PROTECTED] writes: Bernt Hansen [EMAIL PROTECTED] writes: Running a minimal emacs should suppress custom config files: emacs -q -l yourtest.el Added this one to the Clippboard section on new org-tests/index.org in Worg.git. (this section will be temporary...) Something like the above should only be a link (at most) to the emacs manual. Reproducing standard info is bad in the long run in case things in the base product (emacs) change for example. Some kind of regression testing framework would be awesome. Org- mode is large enough that this is almost a necessity to keep things stable and bug-free. It's big and feature-RICH. The nature of OSS means that the community using the product keep it stable and bug free. I dont think the efforts to produce meaningful regression tests would be beneficial in an ever morphing product like org-mode. Clearly my humble opinion on that one :-; I'm a bit clue-free on what the words mean. I think of regression tests as ways of defining the way Things Should Work(TM) so if someone commits as fix to org-mode that works fine on Debian, but breaks on Windows, there's a way for someone on Windows to easily (brainlessly) run the test and report back to the developer. I don't think of org-mode as 'ever morphing'. There are some features that are stable, and should remain so. A regression test would be valuable, presumably, for developers working on a new feature (don't screw up important existing functionality!) I must say I am dubious about this. It means, for the tests to be meaningful, that the output must be a fixed format in base org. I doubt this will ever be the case. The presentation will fluctuate while the core information (dates, schedules periods etc) will remain pretty much constant. I agree that the presentation is fluid and it was hard for me to imagine how to test it in an automated way. But I disagree (I think) in that having a test on the presentation is extremely powerful. I'm thinking that one example of how to write such a test could be copied when someone is trying to explain what they want to happen, and how it is not happening right now. Perhaps such a test would have a very limited useful life. I'd like to think that it could live and be useful for a very long time (that's why thinking about configuration/user customization issues in a test is important). But even if the test is only useful until the bug is fixed (or the feature implemented) , so be it. The majority of bugs that I see are often down to people misusing or using things in the base which are not fully explored. No amount of regression testing can cover things like that unless the regression tests include everyones customisations. Not everyone's customizations, just every customization org-mode has. I mean, if the feature is there, it should work. If the feature is important, it should work in the same way on every platform. The trick -- and I hope it's either solved or easy -- is exactly how to show that the feature works the same way as it should, right now, on this platform If I understand you properly, you're just saying that the user customization creates so much variability in the output, there isn't anyway you could capture all the possible presentations in a single test. Well, yeah, so user customizations need to be left out of the test case. Except that in the scenario I'm thinking of where someone (a mere mortal using org-mode, with limited lisp knowledge) is trying to report a bug, user customization will be critical. My first thought was asking org-mode to dump all the customization variables it has into a buffer (as buffer-local variables maybe?) would help someone else recreate the bug. (By starting org-mode with NO customizations, except what's found in the test case) Or are you suggesting that the source of presentation variation is due mostly to OS, Emacs version, or something else beyond the scope of our tests, so they couldn't be replicated on another machine? Do I think regression testing is important? Yes - in certain environments. But every time Carsten, you, myself or anyone else fires up org-mode we are already doing just that. -- Inventor: A person who makes an ingenious arrangement of wheels, levers and springs, and believes it civilization. ~Ambrose Bierce, The Devil's Dictionary ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
Hi Richard,
Richard Riley [EMAIL PROTECTED] writes:
Added this one to the Clippboard section on new org-tests/index.org in
Worg.git. (this section will be temporary...)
Something like the above should only be a link (at most) to the emacs
manual. Reproducing standard info is bad in the long run in case things
in the base product (emacs) change for example.
On the long run, yes. I just added this section, to start the page. And
since it's just STARTing... this was in the first or second reaction
that came in. Feel free to edit ('go wild' ;-))!
I wouldn't bother to have several pages like this one will be (it's
supposed to be an index on the long run), each covering another way of
testing.
I just meant to take _ACT_. Scepticism is a good thing, as long as it
doesn't stop you from doing.
Some kind of regression testing framework would be awesome. Org-mode is
large enough that this is almost a necessity to keep things stable and
bug-free.
It's big and feature-RICH.
The nature of OSS means that the community using the product keep it
stable and bug free. I dont think the efforts to produce meaningful
regression tests would be beneficial in an ever morphing product like
org-mode. Clearly my humble opinion on that one :-;
We do test our software by using it. But the bug in the HTML-exporter
that Carsten has fixed two days ago, was introduced in early September
and would be in 6.10, which is supposed to be in the emacs 23 release.
A very simple test plan would have revealed it.
Maybe something can be put together from the git testing framework and
use of emacs -batch to process test org files and verify the output is
as expected (with diff or some other tool).
Hey, diff is a good idea!!
I didn't take the verification of the output into account yet :-)
I just pushed a change of Worgs start page, and added a directory
'org-tests'. I've placed an index.org there, which now is just a
collection of ideas (I'm on my day job, so I can't really work on it
now).
Don't know how often the git repo is published.
Bernt and Ben, are you 'worgers' allready?
Do you think it makes sense to add snippets and ideas to the new page in
Worg? I think while the list great to exchange ideas, it's good to have
a place, where all those ideas are destilled to one-liners.
I must say I am dubious about this. It means, for the tests to be
meaningful, that the output must be a fixed format in base org.
Why? If the test bails out 'ERROR', I will have to look for the
reason. If the format changed in a legal way = adjust the test.
I doubt
this will ever be the case. The presentation will fluctuate while the
core information (dates, schedules periods etc) will remain pretty much
constant.
The majority of bugs that I see are often down to people misusing or
using things in the base which are not fully explored. No amount of
regression testing can cover things like that unless the regression
tests include everyones customisations.
Yes, because Carsten add features en masse :-)
I see the testing differently. In the first place we need THINK of
testing.
New Org-revision out?
Ahh, OK, I have to the HTML-exports, I want to be working.
To do this, I need several different setups, several different data
directories (Org-files) and an easy way to test, that doesn't eat my
time and gives a result. The quality may vary, but ERRORs will be
detected for shure.
Not one or mounths later - immediately.
If someone installs emacs 23, tries to export to HTML and gets an
error
Do I think regression testing is important? Yes - in certain
environments. But every time Carsten, you, myself or anyone else fires
up org-mode we are already doing just that.
Yes, but we can do better, easier and more complete.
--
Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover
Tel.: +49 (0)511 - 36 58 472
Fax: +49 (0)1805 - 233633 - 11044
mobil: +49 (0)173 - 83 93 417
Email: [EMAIL PROTECTED], [EMAIL PROTECTED]
Http: www.emma-stil.de
___
Emacs-orgmode mailing list
Remember: use `Reply All' to send replies to the list.
Emacs-orgmode@gnu.org
http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
Bernt Hansen [EMAIL PROTECTED] writes: Some kind of regression testing framework would be awesome. Org-mode is large enough that this is almost a necessity to keep things stable and bug-free. Maybe something like this: http://www.emacswiki.org/emacs/ElUnit ? -- +---+ | Jason F. McBrayer[EMAIL PROTECTED] | | If someone conquers a thousand times a thousand others in | | battle, and someone else conquers himself, the latter one | | is the greatest of all conquerors. --- The Dhammapada| ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
Sebastian Rose [EMAIL PROTECTED] writes: Bernt Hansen [EMAIL PROTECTED] writes: Running a minimal emacs should suppress custom config files: emacs -q -l yourtest.el Added this one to the Clippboard section on new org-tests/index.org in Worg.git. (this section will be temporary...) Something like the above should only be a link (at most) to the emacs manual. Reproducing standard info is bad in the long run in case things in the base product (emacs) change for example. Some kind of regression testing framework would be awesome. Org-mode is large enough that this is almost a necessity to keep things stable and bug-free. It's big and feature-RICH. The nature of OSS means that the community using the product keep it stable and bug free. I dont think the efforts to produce meaningful regression tests would be beneficial in an ever morphing product like org-mode. Clearly my humble opinion on that one :-; Maybe something can be put together from the git testing framework and use of emacs -batch to process test org files and verify the output is as expected (with diff or some other tool). Hey, diff is a good idea!! I didn't take the verification of the output into account yet :-) I just pushed a change of Worgs start page, and added a directory 'org-tests'. I've placed an index.org there, which now is just a collection of ideas (I'm on my day job, so I can't really work on it now). Don't know how often the git repo is published. Bernt and Ben, are you 'worgers' allready? Do you think it makes sense to add snippets and ideas to the new page in Worg? I think while the list great to exchange ideas, it's good to have a place, where all those ideas are destilled to one-liners. I must say I am dubious about this. It means, for the tests to be meaningful, that the output must be a fixed format in base org. I doubt this will ever be the case. The presentation will fluctuate while the core information (dates, schedules periods etc) will remain pretty much constant. The majority of bugs that I see are often down to people misusing or using things in the base which are not fully explored. No amount of regression testing can cover things like that unless the regression tests include everyones customisations. Do I think regression testing is important? Yes - in certain environments. But every time Carsten, you, myself or anyone else fires up org-mode we are already doing just that. -- Inventor: A person who makes an ingenious arrangement of wheels, levers and springs, and believes it civilization. ~Ambrose Bierce, The Devil's Dictionary ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
[EMAIL PROTECTED] (Jason F. McBrayer) writes: Bernt Hansen [EMAIL PROTECTED] writes: Some kind of regression testing framework would be awesome. Org-mode is large enough that this is almost a necessity to keep things stable and bug-free. Maybe something like this: http://www.emacswiki.org/emacs/ElUnit ? The author of elunit (Phil) has abandoned it in favor of ert. ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] Re: How you can help
On Oct 23, 2008, at 6:19 PM, Bernt Hansen wrote: Sebastian Rose [EMAIL PROTECTED] writes: Don't know how often the git repo is published. Bernt and Ben, are you 'worgers' allready? I think it's published hourly (maybe) and yes I have access to Worg. The webserver pulls changes from the Worg git repo on the full hour and publishes these changes on the half hour. So changes will appear on the web after 30-90 minutes. If necessary, this procedure can be upgraded. - Carsten ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
