Re: [Orgmode] How you can help
Avdi Grimm wrote: A few points, from someone with a decent amount of testing background: * As someone who has contributed to OS projects, the lack of a pre-existing set of regression tests in org-mode is actually *the* most significant blocker to my getting involved in org-mode development. I do all of my coding - both professional and personal - in the context of tests, and not having the existing framework means that before I can start working on new features I first need to spend time yak-shaving on testing infrastructure. * I think there's a lot of over-thinking going on here. Here's the test-first coding discipline in a nutshell: 1. Identify a problem/missing feature. 2. Write a test (possibly using a unit-testing framework to help) which will pass when the bug has been fixed or feature added. This can be as simple as calling a function and validating its return value. 3. Run the test. Verify it FAILS. 4. Write code to make the test PASS. 5. Refactor, if you introduced any code duplication in step 4. 6. Run all the tests, to make sure you didn't break anything else. 7. Commit. If someone would be so kind as to identify a small bug or feature, I would be happy to demonstrate this workflow in the form of code, time permitting. I've just started reading and entirely agree that unit testing org-mode would be a good thing. It'd also help people like myself familiarise themselves with elisp, org-mode's internals and the expectations of certain features. Anyway, given that nobody seems to have found you a simple definitive bug, here's a suggestion: Why not look through the git repo for a trivial bug carsten fixed in the past, branch the repo at the point the bug existed and write a regression test for it? Then test the fix against the commit that fixes it. This might also prove a nice way for other people to introduce tests into org-mode. By targetting areas where we have had bugs previously, we can build up coverage, and develop regression tests for known (historical) issues. Source control systems (and especially git) are wonderful things. A really nice feature of git which might help moving forward is git bisect. git bisect is typically used to find the commit that introduced a particular bug or regression. The typical workflow is this: 1. You realise there is a bug on a branch head, but you don't know when it was been introduced. 2. With git bisect you mark the commit that you know has the bug as 'bad'. 3. With git bisect you mark the last known good commit. 4. You run git bisect, and it will recursively find you a commit in the middle of your good and bad commits, and ask you whether it is good or bad. It proceeds in a binary-chop manner, until it finds the precise commit that introduced the bug. The awesome thing is that you can completely automate this process by writing a program/test that can test for the presence of this bug! If it were desirable to make use of this feature, then it might be better if the unit tests were written in a different git repo (perhaps also referenced as a git submodule so the tests can also be located inside the org-mode repo for convenience). Doing this would ensure that the tests can exist outside of org-mode, allowing you to easily run bisect. Avoiding the issue of bisecting to a point before the test existed. Anyway, it's just a suggestion! And all the best on introducing some tests! :-) R. ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Avdi Grimm [EMAIL PROTECTED] writes: A few points, from someone with a decent amount of testing background: Jippie! Please stay with us for a few days :-D [...] As someone with also a decent amount of testing background, I'd like to answer also. Did you work with unit-testing frameworks for elisp already? Which one? I co-wrote regress.el (Wayne Mesard wrote the first version) and I wrote rtest. I've written some testing for allout.el before I switched over to org-mode (which is very nice, BTW, Carsten, and provides pretty much everything I wish-listed for allout mode). Of particular interest for testing org-mode, I wrote mockbuf.el, which mocks files and buffers so that tests can deal with files without changing anything permanently. Recommendations? Hard for me to say objectively. I prefer my rtest but it's in flux right now. I wouldn't recommend the current public version, rtest 3.0 I've designed and half written a better version but I can't really ask you to wait until I get around to making a better version of rtest. There is a development version that works and does nice things; it's messy as hell right now and I'm embarrassed about that. It isn't public but I could put it up if you'll forgive its messy state, It has mockbuf, which I found indispensable in writing tests for allout.el. In fact, I mostly developed it for that, though all the allout-specific code is elsewhere. Also useful for writing tests: safe-equal.el, caselet.el, and select.cl, all in the ancillary directory. I would steer clear of frameworks that provide many of synonyms for equality tests, inequality tests, etc. You already know how to write (not (equal ...)) in elisp. Why learn the name and syntax of something like (shouldnt-be-equal A B)? Having a lot of that makes a package LOOK heftier but doesn't do anything more for you. IMO analyzing test failures should be done by inspecting the test form, not by making you write things in non-standard elisp that could be written in standard elisp. Using ert would mean requiring ewoc, which means tests wouldn't work on emacs 21. Too bad, because it has some appeal. Could say something about the effort to get started with such a framework? For rtest, right now you just need (a) the development version, (b) unzip it in your emacs-site-lisp or set the paths and autoloads manually if you prefer, (c) include the following in a file to be tested: (when (not (fboundp 'rtest:deftest)) (defmacro rtest:deftest (rest dummy)) (defmacro rtest:if-avail (rest dummy))) Then write tests as: (rtest:deftest function-name (Docstring (test-form) ;;Return a boolean ) ;;Repeat as wanted ) That will get you started, but really you'd want to get familiar with the stuff in mockbuf.el - which is so messy right now that I'm embarrassed about it, because it contains both old and new versions of many things in flux. Other advice: Make a directory to hold short examples of files that illustrate before and after states for the various transformations. Each unit (eg org-agenda.el, org-archive.el etc) should have its own subtree of examples. One thing I'd do early is build the following test helper functions: A macro that runs a test body inside a let of all the configuration options set to expected values. A function to compare vector-style strings. `equal' doesn't suffice. rtest has, in ancillary/safe-equal.el, safe-equal-compare-objects and safe-equal-compare-vectors, which IIRC suffice for this. The ability to write only the visible parts to a separate buffer was indispensable in testing allout's folding. You already have it in org-export-visible with the type that keeps the buffer, ?\ . Then you can just compare contents. You will likely have a lot of what I call output tests, tests of code that has the property that if the output looks right, it is right. IMO with output tests, you need to make it easy on yourself. TDD is all well and fine, but not if it makes you repeat yourself. If there is available output that looks right, capture it and use it as a base for comparison. That's not pure but it's far easier and it works right. If the available output isn't yet right, capture it anyways and tweak it to make it right - but this rarely happens. I automated that sort of thing for a Prolog test harness I wrote, works fine. You'd just write a test of a special type that produced and captured output. It would originally be ungraded, but you would then grade it, and it would remember the correct output and compare against it. I was planning to add the same to rtest and I partly did, but never finished it. Still on my todo list. Can we add it without changing Org's code? If understand 2. correctly - yes? rtest's tests like to be in the same file as the code. If you use the form above, the file still compiles and runs even when rtest is not available. It's
Re: [Orgmode] How you can help
On 2008-Oct-23, at 17:42, Avdi Grimm wrote: If someone would be so kind as to identify a small bug or feature, I would be happy to demonstrate this workflow in the form of code, time permitting. Ok, here's your chance. This is something that has bothered me for quite some time, but I've never been able to reliably reproduce the problem. And it's such a small issue. Sometimes, when it looks like my cursor is at the end of a headline and I hit tab, the headline does not cycle. Normally it does. It hit me that the point was not on the headline, it only looked like it. So here's the sample test.org file * First Headline Some body text * Second Headline (goto-char 17) ;that's the end of the first line (org-cycle) At this point the buffer looks like: * First Headline... * Second Headline (goto-char 25) (org-cycle) I think the buffer should look like this * First Headline Some body text * Second Headline but it still looks like: * First Headline... * Second Headline * ABOUT THIS BEHAVIOR I should be honest here. This may not be a bug. As I created this test data, it became a bit more clear that exactly where the cursor is matters a lot here. I *promise* that I've seen this behavior even when the cursor shows up before the ellipses, but in the test case I set up here, I could not make that happen. I also found that if I used the key-chord M-x goto-char RETURN 25, everything worked fine. The cursor stayed in front of the ellipses and the tab key worked. But when I used M-: (goto-char 25) the cursor moved to the same line as the ellipses, but after them. and the TAB key stopped 'working'. I finally figured this out while playing with git. I switched branches at the command line and needed to perform a 'revert-buffer'. This left the cursor before the ellipses, but unable to properly (org- cycle) using the TAB key. More experimenting I *think* the revert-buffer command tries to keep point close to the same place, and the org buffer had automatic folding. The bad thing about this for me is that I'll hit the TAB key four times trying to make sure it's not just an empty tree. Meanwhile, (org-cycle) has indented the first line of the body, but hasn't shown me the text I'm changing. Then I get confused about why the buffer needs saving, probably 10 minutes later when I've forgotten all about hitting the tab key. But I'm way out of my depth to try to understand the relationship between (revert-buffer) (org-cycle) the arrow keys, and all. I just hit M- or S-TAB and try again. Or C-a, which sometimes works. Sometimes I grab the mouse as a last resort. * ABOUT TESTING ISSUES Just from this example, I'm eager to understand: 1. How can we differentiate between where the cursor appears and the value of point 2. How can we tell what the users sees on the screen versus what the buffer contains So the value of a testing framework is that this: if I'm going to announce to the world that I can't get something to work, (like the tab key), I'm going to make darn sure it's a real problem. I'm going to spend the time to create a small sample file, that reliable creates the problem, and I'm going to try a few different scenarios. But if I start to get lost or confused about which settings I've fiddled with or what is supposed to happen, I'll wander away from the problem and I won't submit the bug. We all lose in that scenario. But if the testing framework exists, and it is lightweight enough for novice emacs users (advanced enough to use M-x but not advanced enough to read lisp) then maybe I would have used it for this example. -Ben ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
On Fri, Oct 24, 2008 at 2:33 PM, Ben Alexander [EMAIL PROTECTED] wrote: Ok, here's your chance. This is something that has bothered me for quite some time, but I've never been able to reliably reproduce the problem. And it's such a small issue. Thanks! I'll try to take a look tonight or this weekend. So the value of a testing framework is that this: if I'm going to announce to the world that I can't get something to work, (like the tab key), I'm going to make darn sure it's a real problem. I'm going to spend the time to create a small sample file, that reliable creates the problem, and I'm going to try a few different scenarios. But if I start to get lost or confused about which settings I've fiddled with or what is supposed to happen, I'll wander away from the problem and I won't submit the bug. We all lose in that scenario. But if the testing framework exists, and it is lightweight enough for novice emacs users (advanced enough to use M-x but not advanced enough to read lisp) then maybe I would have used it for this example. For the record, I normally would never expect the user to submit tests. A description of the problem and some sample data is all I expect. Writing the a test to reproduce the behavior is the programmer's responsibility as far as I'm concerned. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
On Fri, Oct 24, 2008 at 14:44, Avdi Grimm [EMAIL PROTECTED] wrote: For the record, I normally would never expect the user to submit tests. A description of the problem and some sample data is all I expect. Writing the a test to reproduce the behavior is the programmer's responsibility as far as I'm concerned. That's the best part of open source software! A surprising amount of the time, the user IS a programmer, and thus is able to provide tests. And it's not that we should require anyone to provide tests before they report a bug, but they should be required to outline the steps to reproduce the behavior, so any programmer could take 5 minutes and quickly whip up a test. ps: I almost have a baby ert framework for org-mode. It doesn't do anything yet, but the design is you write a test, and add it to the org-test-hook to be run by org-test-all. -- . : [ + carpe diem totus tuus + ] : . ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
[Orgmode] How you can help
Hi Org users, I need to get control over the time I spent on developing Org-mode. Recently, I have again worked too hard on it, spending more time than I should, in order to get 6.9 and 6.10 out and to seize the chance to get the best possible version into Emacs 23.1. However, this is getting out of control, and I am now putting myself a hard limit of 1 hour per day, clocked by Org-mode, which will clearly cut in on my development speed and posting rate. Here is how you can help me to make the most of that one hour. 1. If you report a bug, please try to do as much work as you can yourself. Before you send it, think if you have collected all the information you can. There are great examples of good bug reports on the list already. The best bug reports are, of course, those that are accompanied by a patch. 2. If you are reading the list and see bug reports, consider putting in 10 minutes, trying to reproduce this problem yourself, and maybe adding information that might be useful for me to track down that bug. Again, this is already happening, I am only trying to encourage this type of behavior. 3. I have recently spent much time on fixing bugs in parts of Org that I did not write myself, so this is taking much more time than usually. If you know Emacs Lisp, and I know there are a number of excellent Lisp programmers on this group, consider adopting one of the following subsystems. By adoption, I mean that you make it your mission to deeply understand this part of Org, so that *you* will be in the position to fix bugs, taking that off my shoulders. - org-publish.el. I think the biggest bugs are out now, but I am sure this system can be improved quite a bit. If you are that Lisp programmer trying to take up this task, consider teaming up with Sebastian Rose who is a great guy, has quite some understanding of that system and good ideas about it, but just is not enough of a Lisp programmer to really take that on. - org-export-latex.el. This is a tough one because you need to know both LaTeX and Lisp. And the code is complex, in part because Org-mode allows to write LaTeX is a relatively lazy way. Bastien has done a great great job capturing this into an exporter. However, there are still problems and bugs, some came up recently on the list. And Bastien currently does not have the time to contribute consistently. As a result, I have started to fix some bugs, but this is really eating too much of my time. This subsystem can also use feature additions, for example better handling of image insertion, maybe with captions. I have ideas about this, so talk to me if you want to help out. 4. Try to answer as many messages on the mailing list as you can. I have been trying hard to make sure that each and every reasonable question on the list (and this is really the only kind we get in this amazing community) will be answered. Doing this still takes a significant fraction of my Org-mode development time. I will clearly spend less time on this in the future, in this way also giving others more time to answer. Again, there are already quite a few people who regularly *answer* questions, and all I am trying to do here is encouraging this activity. 5. Help developing the Org-mode FAQ by adding useful information to it yourself. All you need to do is to get acces to Worg, which will help getting to know git in the process. Worg is meant to be user edited, so please go wild and add information and links at will. Thanks to you all, for your understanding and help. - Carsten ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
On Thu, Oct 23, 2008 at 1:05 PM, Carsten Dominik wrote: Hi Org users, I need to get control over the time I spent on developing Org-mode. Recently, I have again worked too hard on it, spending more time than I should, in order to get 6.9 and 6.10 out and to seize the chance to get the best possible version into Emacs 23.1. However, this is getting out of control, and I am now putting myself a hard limit of 1 hour per day, clocked by Org-mode, which will clearly cut in on my development speed and posting rate. Here is how you can help me to make the most of that one hour. 1. If you report a bug, please try to do as much work as you can yourself. Before you send it, think if you have collected all the information you can. There are great examples of good bug reports on the list already. The best bug reports are, of course, those that are accompanied by a patch. 2. If you are reading the list and see bug reports, consider putting in 10 minutes, trying to reproduce this problem yourself, and maybe adding information that might be useful for me to track down that bug. Again, this is already happening, I am only trying to encourage this type of behavior. 3. I have recently spent much time on fixing bugs in parts of Org that I did not write myself, so this is taking much more time than usually. If you know Emacs Lisp, and I know there are a number of excellent Lisp programmers on this group, consider adopting one of the following subsystems. By adoption, I mean that you make it your mission to deeply understand this part of Org, so that *you* will be in the position to fix bugs, taking that off my shoulders. - org-publish.el. I think the biggest bugs are out now, but I am sure this system can be improved quite a bit. If you are that Lisp programmer trying to take up this task, consider teaming up with Sebastian Rose who is a great guy, has quite some understanding of that system and good ideas about it, but just is not enough of a Lisp programmer to really take that on. - org-export-latex.el. This is a tough one because you need to know both LaTeX and Lisp. And the code is complex, in part because Org-mode allows to write LaTeX is a relatively lazy way. Bastien has done a great great job capturing this into an exporter. However, there are still problems and bugs, some came up recently on the list. And Bastien currently does not have the time to contribute consistently. As a result, I have started to fix some bugs, but this is really eating too much of my time. This subsystem can also use feature additions, for example better handling of image insertion, maybe with captions. I have ideas about this, so talk to me if you want to help out. 4. Try to answer as many messages on the mailing list as you can. I have been trying hard to make sure that each and every reasonable question on the list (and this is really the only kind we get in this amazing community) will be answered. Doing this still takes a significant fraction of my Org-mode development time. I will clearly spend less time on this in the future, in this way also giving others more time to answer. Again, there are already quite a few people who regularly *answer* questions, and all I am trying to do here is encouraging this activity. 5. Help developing the Org-mode FAQ by adding useful information to it yourself. All you need to do is to get acces to Worg, which will help getting to know git in the process. Worg is meant to be user edited, so please go wild and add information and links at will. Thanks to you all, for your understanding and help. It is quite a challenge to try out and understand numerous features and facilities already in Org and the pace at which the new changes were coming was unreal and super-human. The backlog of things I wanted to try, understand and absorb in my workflow was consistently increasing. This is not a complaint; I am having so much fun. *BUT* I do not want my favourite developer of my favourite package in my favourite editor to burn out. Please do take it easy. Unfortunately, I can not help with #3 (adopting a subsystem) since I am Lisp-challenged but with rest I will. Sincere thanks for all your time, effort and sacrifices. -- Manish ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Should we put a page on worg by this name (see subject of this thread)? We could show, how to turn on debugging, write a good bug report, link to good elisp tutorials and describe how to use the elisp debugger in emacs for simple debugging. If no one stops me, I'll do it these days. As or me, a page like that would be helpfull. I'll be back for report - maybe one of the lisp freaks could add some details later on? Regards, Sebastian Carsten Dominik wrote: Hi Org users, I need to get control over the time I spent on developing Org-mode. Recently, I have again worked too hard on it, spending more time than I should, in order to get 6.9 and 6.10 out and to seize the chance to get the best possible version into Emacs 23.1. However, this is getting out of control, and I am now putting myself a hard limit of 1 hour per day, clocked by Org-mode, which will clearly cut in on my development speed and posting rate. Here is how you can help me to make the most of that one hour. 1. If you report a bug, please try to do as much work as you can yourself. Before you send it, think if you have collected all the information you can. There are great examples of good bug reports on the list already. The best bug reports are, of course, those that are accompanied by a patch. 2. If you are reading the list and see bug reports, consider putting in 10 minutes, trying to reproduce this problem yourself, and maybe adding information that might be useful for me to track down that bug. Again, this is already happening, I am only trying to encourage this type of behavior. 3. I have recently spent much time on fixing bugs in parts of Org that I did not write myself, so this is taking much more time than usually. If you know Emacs Lisp, and I know there are a number of excellent Lisp programmers on this group, consider adopting one of the following subsystems. By adoption, I mean that you make it your mission to deeply understand this part of Org, so that *you* will be in the position to fix bugs, taking that off my shoulders. - org-publish.el. I think the biggest bugs are out now, but I am sure this system can be improved quite a bit. If you are that Lisp programmer trying to take up this task, consider teaming up with Sebastian Rose who is a great guy, has quite some understanding of that system and good ideas about it, but just is not enough of a Lisp programmer to really take that on. - org-export-latex.el. This is a tough one because you need to know both LaTeX and Lisp. And the code is complex, in part because Org-mode allows to write LaTeX is a relatively lazy way. Bastien has done a great great job capturing this into an exporter. However, there are still problems and bugs, some came up recently on the list. And Bastien currently does not have the time to contribute consistently. As a result, I have started to fix some bugs, but this is really eating too much of my time. This subsystem can also use feature additions, for example better handling of image insertion, maybe with captions. I have ideas about this, so talk to me if you want to help out. 4. Try to answer as many messages on the mailing list as you can. I have been trying hard to make sure that each and every reasonable question on the list (and this is really the only kind we get in this amazing community) will be answered. Doing this still takes a significant fraction of my Org-mode development time. I will clearly spend less time on this in the future, in this way also giving others more time to answer. Again, there are already quite a few people who regularly *answer* questions, and all I am trying to do here is encouraging this activity. 5. Help developing the Org-mode FAQ by adding useful information to it yourself. All you need to do is to get acces to Worg, which will help getting to know git in the process. Worg is meant to be user edited, so please go wild and add information and links at will. Thanks to you all, for your understanding and help. - Carsten ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Sebastian Rose wrote: Should we put a page on worg by this name (see subject of this thread)? We could show, how to turn on debugging, write a good bug report, link to good elisp tutorials and describe how to use the elisp debugger in emacs for simple debugging. If no one stops me, I'll do it these days. As or me, a page like that would be helpfull. I'll be back for report - maybe one of the lisp freaks could add some details later on? Hm - there is a link to the 'Feedback' section of the manual (http://orgmode.org/org.html#Feedback) on http://orgmode.org/worg/org-contribute.php but 1. It's hard to find. One has find worg (http://orgmode.org/worg/), and then click 'How to contribute to Org?' to get to http://orgmode.org/worg/org-contribute.php where the link is. I think I wouldn't follow the link 'How to contribute to Org', if I was to report a bug (maybe it's my bad english, but I lots of people whos english is even worse). 2. While the 'Feedback' section in the manual is as good as the rest of it, it's short and does, for good reasons, not provide more information, than absolutely needed. Suggestions: 1. on Worgs startpage (http://orgmode.org/worg/): add an extra section (as the FIRST one) 'How you can help'. 2. Under that headline, put the link to the 'Feedback' section of the manual. 3. Add more links to one or more pages for tips on debugging etc. 4. Add subdirectory for those debugging tips. We could also add things like coding conventions, notes about the APIs in org etc. to that directory. Please comment... 5. I also think of little packages for testing parts of org. Example XHTML-export: When the HTML-exporter didn't work lately, I set up a little directory or testing. On top of that tree I had an Org-file with [[elisp:]] links to click on, like [[elisp:(setq debug-on-error t)]] or [[elisp:(debug-on-entry 'org-export-org-to)]] and one for loading a setup.el in the same directory (primaryly to adjust org-publish-project-alist - in principle, everything could be setup in that file, but needed two different setups for these tests). Now I can just open that Org-file, click on two or three links and get a backtrace, if something is wrong. It's so simple to test the exporter this way, that it takes 30 seconds or even less (on failure :-) With simple trees like that (eventually downloadable as *.tar.bz2) one may test a module with all the little corner cases, which might not be in the Org-files the tester uses for his daily work and without risk. If we encounter a new corner case, we could just add it to the test packages Org-files and document it in the packages index.org (or README.org) to prevent a regression. When working on a module, one could programm against those tests. Creating and maintaining those test packages is a simple thing to do. All it requires is knowledge of USING org and just a tiny little extra. Hence I suppose this work should be done by the community/worgers. It will be much more fun to add tested patches (and more secure to create one)! Regards, Sebastian Carsten Dominik wrote: Hi Org users, I need to get control over the time I spent on developing Org-mode. Recently, I have again worked too hard on it, spending more time than I should, in order to get 6.9 and 6.10 out and to seize the chance to get the best possible version into Emacs 23.1. However, this is getting out of control, and I am now putting myself a hard limit of 1 hour per day, clocked by Org-mode, which will clearly cut in on my development speed and posting rate. Here is how you can help me to make the most of that one hour. 1. If you report a bug, please try to do as much work as you can yourself. Before you send it, think if you have collected all the information you can. There are great examples of good bug reports on the list already. The best bug reports are, of course, those that are accompanied by a patch. 2. If you are reading the list and see bug reports, consider putting in 10 minutes, trying to reproduce this problem yourself, and maybe adding information that might be useful for me to track down that bug. Again, this is already happening, I am only trying to encourage this type of behavior. 3. I have recently spent much time on fixing bugs in parts of Org that I did not write myself, so this is taking much more time than usually. If you know Emacs Lisp, and I know there are a number of excellent Lisp programmers on this group, consider adopting one of the following subsystems. By adoption, I mean that you make it your mission to deeply understand this part of Org, so that *you* will be in the position to fix bugs, taking that off my shoulders. - org-publish.el. I think the biggest bugs are out now, but I am sure this
Re: [Orgmode] How you can help
Sebastian Rose wrote: 5. I also think of little packages for testing parts of org. I'm curious if you or someone else has any ideas for writing automated tests for org-mode. I haven't the foggiest idea how someone would write a test for the parts of org that control what is displayed on the screen. I mean, when the bug is 'it doesn't look right' how can you tell? Perhaps the git repository should have a small collection of small org- mode files that reproduce certain bugs? If there were some examples of how to create such a test, then perhaps bug reporters would find it much easier to create them. I do see some confusing issues due to different configuration files. So creating a test file might involve making sure org-mode doesn't read any configuration (how do you do that?) and possible asking org- mode to extract all the configuration variables it has right now and dump them into a test file (...and how do you do that?) ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Hi Ben, I cc'ed the list. The tests I described in my email to the list are not automated. The reason for that is my lack of (e)lisp knowledge. BUT, they where easy to handle for non programmers. I think the little test will make it to the worg site this week, when all private data is removed. You could take a look at it by then. And: improve, improve, improve... :-) If you know of someone who knows how to do automated tests in elisp, or some technique, package, whatever, please post it to the list, so we all can take a look at it and comment-decide something. This is _highly_ _appreciated_. Ben Alexander [EMAIL PROTECTED] writes: Sebastian Rose wrote: 5. I also think of little packages for testing parts of org. I'm curious if you or someone else has any ideas for writing automated tests for org-mode. I haven't the foggiest idea how someone would write a test for the parts of org that control what is displayed on the screen. I mean, when the bug is 'it doesn't look right' how can you tell? Believe me, my idea is the foggiest of all possible ideas ;-) (so where my ideas of JavaScript some mounth ago), so I won't be of much help, I fear. Perhaps the git repository should have a small collection of small org- mode files that reproduce certain bugs? If there were some examples of how to create such a test, then perhaps bug reporters would find it much easier to create them. YES! Exactly. Every corner case an Org-file, every bug an Org-file. _DATA_ for testing is something, everyone _can_ provide. But git later, yes, maybe. Since this would need Carsten to think and act on this, I feel Worg is a nice place to start the first expieriments. We need Carstens power for other things (when will Org-mode finaly wash my car? It's repeater-TODO, but nothing happens!!!) Basically, I'd try to keep the the testing as simple as possible, and try to get elisp programmers to help with the code. We should try to keep the hurdles for testers as humble (?) as possible, and put all that's needed to be helpfull on one page in worg. I recently discovered the very unautomated `(print object)' in the elisp reference. Not everything can be done automated, maybe, but if I would have known this stupid `print' function, I would know more about elisp and some files in org already. And it would have been faster and easier to create the two minor patches I wrote. This is, where the 'links to elisp tutorials', 'tipps and tricks', 'emacs debugger' come handy. Willingness to help is no problem, as the Org-community reveals. As for me, it's often a lack of time and/or knowledge, that prevents it. And the aim of all this is to help helping, in means of good and detailed bug reports in the first place. I do see some confusing issues due to different configuration files. So creating a test file might involve making sure org-mode doesn't read any configuration (how do you do that?) and possible asking org- mode to extract all the configuration variables it has right now and dump them into a test file (...and how do you do that?) True. Hm - to test without configuration, we already have `emacs -q'. Dumping the variables is just a list of (print var) statements. A (quite) complete list of variables could be extracted with grep? e.g.: grep -r defvar lisp/ Once we have such a list, we could set/reset some/all variables. This will not be perfect, but could work reasonable well. And yes, it would indeed be nice, to have elisp to reliably reset emacs/org configuration, so one could do several different test in sequence. Preferably they would _always_ run and dump all errors to some file or buffer, even if one or more tests fail. I think elsip provides funtionalities to handle those errors and skip to the next test. This would be a huge step. But still, we have to test with different _DATA_ too! The test I described is just a quick hack I did to do the testing, while Carsten was bug-hunting on the other end of my internet connection. The XHTML-export test was much easier to do, then some other tests we might need. In the end, it was a test, automated or not. And I had to do it, because I'm one out of a few who use org to maintain a web-site (locally in my case) and export an entire directory tree with lots of _DATA_, use org-info.js, `#+SETUPFILE' lines, etc. , I believe. This became obvious in this case. I was the one who even noticed the bug, which means, no one else was using recursive XHTML export for a while, or their setup didn't reveal that bug. So clearly I was the one to provide some support for this very part of the system. I can't rely on the assumption, that some maintainer has all possible setups at hand all the time (maybe this was possible years ago). While the testing of the HTML-export was quite simple, it requires a lot of _DATA_. Namly files and directories + different setups, per file setups, #+SETUPFILE, images, with/without org-info.js, extra styles set or not... to
Re: [Orgmode] How you can help
On Thu, Oct 23, 2008 at 7:50 PM, Sebastian Rose wrote: [snip] If you know of someone who knows how to do automated tests in elisp, or some technique, package, whatever, please post it to the list, so we all can take a look at it and comment-decide something. This is _highly_ _appreciated_. We could start here: http://www.emacswiki.org/emacs-fr/UnitTesting#toc2 -- Manish ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting Unfortunately for me, I can't tell if Emacs comes with any builtin framework already, so I downloaded one of the many options listed on that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el The personal issue I have is that I'm on a Mac, using Aquamacs, and the command line version of emacs is a different binary, so there might be trouble when a test passes at the command line, but fails where it matters to me. I don't even make sure that org-mode is up to date at the command line. I thought it wasn't, but I just checked and now it is. Plus, I don't really understand internals of emacs (like basic internals: I understand point and mark, buffer and file, but not transient mark, indirect buffer, symbols vs strings, window vs tab vs frame) The tutorial I'd need to write a test is one which lays out code I could copy and paste to do the following * setup the test environment - create a test directory - create a sample test.org file - put the cursor in a particular place * run the command we need to test - hit the 'TAB' key, or C-c C-c (some folks might need to be reminded how to find out exactly what command is actually being run when you hit a keystroke. And some of me might need to be told what lisp-code to use when the keystroke runs different commands at different places in a file) - reformat a table - clock in/out - create the agenda - export .html .ics .dvi file * How do we specify the correct result??? - check that the headline folded properly. What's the lisp code for getting the folded string as displayed? - check that the cursor is where it should be? especially when the cursor is near elipses... - check that the agenda is built properly. What's the lisp code for getting the agenda as a string? - check that the exported files are correct. Maybe the right suggestion is to run the export on two different files, so the test can focus on the 'diff' between them. That way different people who run the same test on different hosts can get the same result. More experienced folks must suggest whether it's better to have the expectation of a test be specified as a string in the test code or as a separate file in the test directory. It might depend on context, so feel free to lay out three or four cases. If even one kind of test for each of these 'cases' exists, hopefully we can encourage people to report bugs with the test. I mean, if people are already going through the effort to create the small sample org file to demonstrate their bug, and it's only a short (obvious) step to translating their english description of what should happen into equivalent lisp code that seems like a big temptation for someone to do it. Plus it might help them convince themselves that it really is a bug, and not that they are going crazy. I gotta move on for the day here, but I'll keep thinking a bit about this. On 2008-Oct-23, at 15:20, Sebastian Rose wrote: Hi Ben, I cc'ed the list. The tests I described in my email to the list are not automated. The reason for that is my lack of (e)lisp knowledge. BUT, they where easy to handle for non programmers. I think the little test will make it to the worg site this week, when all private data is removed. You could take a look at it by then. And: improve, improve, improve... :-) If you know of someone who knows how to do automated tests in elisp, or some technique, package, whatever, please post it to the list, so we all can take a look at it and comment-decide something. This is _highly_ _appreciated_. Ben Alexander [EMAIL PROTECTED] writes: Sebastian Rose wrote: 5. I also think of little packages for testing parts of org. I'm curious if you or someone else has any ideas for writing automated tests for org-mode. I haven't the foggiest idea how someone would write a test for the parts of org that control what is displayed on the screen. I mean, when the bug is 'it doesn't look right' how can you tell? Believe me, my idea is the foggiest of all possible ideas ;-) (so where my ideas of JavaScript some mounth ago), so I won't be of much help, I fear. Perhaps the git repository should have a small collection of small org- mode files that reproduce certain bugs? If there were some examples of how to create such a test, then perhaps bug reporters would find it much easier to create them. YES! Exactly. Every corner case an Org-file, every bug an Org-file. _DATA_ for testing is something, everyone _can_ provide. But git later, yes, maybe. Since this would need Carsten to think and act on this, I feel Worg is a nice place to start the first expieriments. We need Carstens power for other things (when will Org-mode finaly wash my car? It's repeater- TODO, but nothing happens!!!) Basically, I'd try to keep the the
Re: [Orgmode] How you can help
On Thu, Oct 23, 2008 at 09:35:46AM +0200, Carsten Dominik wrote: Hi Org users, I need to get control over the time I spent on developing Org-mode. Recently, I have again worked too hard on it, spending more time than I should, in order to get 6.9 and 6.10 out and to seize the chance to get the best possible version into Emacs 23.1. However, this is getting out of control, and I am now putting myself a hard limit of 1 hour per day, clocked by Org-mode, which will clearly cut in on my development speed and posting rate. Carsten, you need a break sometime too. ;] - org-export-latex.el. This is a tough one because you need to know both LaTeX and Lisp. And the code is complex, in part because Org-mode allows to write LaTeX is a relatively lazy way. Bastien has done a great great job capturing this into an exporter. However, there are still problems and bugs, some came up recently on the list. And Bastien currently does not have the time to contribute consistently. As a result, I have started to fix some bugs, but this is really eating too much of my time. This subsystem can also use feature additions, for example better handling of image insertion, maybe with captions. I have ideas about this, so talk to me if you want to help out. As you've discovered I'm willing to test latex issues, but I don't know Lisp. Whoever picks this up can continue to use me for testing. Thanks to you all, for your understanding and help. - Carsten You still rock! -- Russell Adams[EMAIL PROTECTED] PGP Key ID: 0x1160DCB3 http://www.adamsinfoserv.com/ Fingerprint:1723 D8CA 4280 1EC9 557F 66E8 1154 E018 1160 DCB3 ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Date: Thu, 23 Oct 2008 15:55:41 +0100 From: Ben Alexander [EMAIL PROTECTED] Subject: Re: [Orgmode] How you can help To: Sebastian Rose [EMAIL PROTECTED] Cc: emacs-orgmode Org-Mode emacs-orgmode@gnu.org Message-ID: [EMAIL PROTECTED] Content-Type: text/plain; charset=US-ASCII; format=flowed; delsp=yes Well, I was just looking at http://www.emacswiki.org/emacs-fr/UnitTesting Unfortunately for me, I can't tell if Emacs comes with any builtin framework already, so I downloaded one of the many options listed on that page to my local site-lisp directory: http://www.wanglianghome.org/svn/test/test.el The personal issue I have is that I'm on a Mac, using Aquamacs, and the command line version of emacs is a different binary, so there might be trouble when a test passes at the command line, but fails where it matters to me. I don't even make sure that org-mode is up to date at the command line. I thought it wasn't, but I just checked and now it is. Plus, I don't really understand internals of emacs (like basic internals: I understand point and mark, buffer and file, but not transient mark, indirect buffer, symbols vs strings, window vs tab vs frame) Actually, if you want, you *can* run Aquamacs from the command line, but it can be a pain to do it. I figured out how to do this when I was trying to use the Makefile for org-mode. I ended up with the following emacs command-line: EMACS=/Applications/Aquamacs\ Emacs.app/Contents/MacOS/Aquamacs\ Emacs and this line for batchmode compiling. Note that I had to augment the standard emacs command-line -q option with Aquamacs' -Q: BATCH=$(EMACS) -batch -Q -q -eval \ (progn (add-to-list (quote load-path) (expand-file-name \./lisp/\)) \ (add-to-list (quote load-path) \$(lispdir)\)) HTH, r ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
I've been considering buckling down and doing some more ELisp coding, which would mean org-mode coding because I've become ridiculously dependent on org-mode. If I do, I would only do it in a test-first way, because that's the only way I'll write code anymore; so it would definitely be good to know that unit tests will be be accepted into the codebase. Personally, if I did this I think I would like to use Emacs Lisp Expectations: http://www.emacswiki.org/cgi-bin/emacs/EmacsLispExpectations I'm a big fan of Jay Field's work, I like the way the original Expectations for Ruby is structured, and I like the looks of the EL Expectations API. I'm open to other libraries though - so long as SOME unit-testing standard it accepted. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
Ben, I added this to the new page on worg. Please tell me: is it OK for you, if I simply add your ideas there?? What do the others think? Ben Alexander [EMAIL PROTECTED] writes: The tutorial I'd need to write a test is one which lays out code I could copy and paste to do the following * setup the test environment - create a test directory - create a sample test.org file - put the cursor in a particular place * run the command we need to test - hit the 'TAB' key, or C-c C-c (some folks might need to be reminded how to find out exactly what command is actually being run when you hit a keystroke. And some of me might need to be told what lisp-code to use when the keystroke runs different commands at different places in a file) - reformat a table - clock in/out - create the agenda - export .html .ics .dvi file * How do we specify the correct result??? - check that the headline folded properly. What's the lisp code for getting the folded string as displayed? - check that the cursor is where it should be? especially when the cursor is near elipses... - check that the agenda is built properly. What's the lisp code for getting the agenda as a string? - check that the exported files are correct. Maybe the right suggestion is to run the export on two different files, so the test can focus on the 'diff' between them. That way different people who run the same test on different hosts can get the same result. Please, if you find time, add the rest somewhere there? If not, I'll take a closer look tonight and check in what's interesting (which seems to be all of it, actually). http://www.emacswiki.org/emacs-fr/UnitTesting: Does this mean to add code to Org's *.el files? This would mean a lot of work for Carsten :-( OK, once done... But: sh grep -r defun lisp/ | wc -l 1290 Checking in/out of every function makes 2580 places to check. I can't estimate the effort, actually, because I never did Unit testing. I guess this number much higher than needed? Could anyone with testing expierience estimate? While I think automated tests are a great thing to do and I'm happy to see this reaction on the list :-), I still think we should also provide the 'simple' tests without additional dependencies (when will package management make it into emacs...). And I'd like to keep the effort and impact for Carsten as small as possible. He will have to check each and every patch because of feeling, and actually being, responsible for the code. Another question: will the code compile in emacs without the testing framework installed? Probably yes. Should we try to think of those 'simple test packages' as the more _DATA_ part of other tests? This would mean one could code an automated test without the need to create data too, since we have the data seperated. Again, I'd like to keep the hurdles low for everyone involved. When automated testing turns out to be a foolish easy and rock solid, this could change. * Three basic approaches: 1 Simple data and setup packages. 2 Code, that executes org commands and test for correct output, no changes in Org's code needed. 3 Test framework with changes to Org's code. I'd promote a combination from 1 and 2, and see, how far it goes. DATA +--- d1 +--- d2 +--- d3 TESTS +---t-a +---t-b +---t-c Now one would run test t-a over d1, d2, d3. Still a programmer could use d1 to fix a bug in d1 and finally push the fix for automated testing against d2 and d3. bug X tester verifies (is ist bug? how to reproduce...) tester adds X to dX (message - list or programmer) programmer installs dX programmer fixes bugX programmer push (message - list or tester) tester tests against d1, d2, d3 where tester = [human|machine] preferable human (simplicity) Should we track the dependencies between code and tests? Suppose one has worked on a.el. Shure more than one test has to run. Example: org-publish.el contains code common to LaTeX-export, HTML-export... Regards, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: [EMAIL PROTECTED], [EMAIL PROTECTED] Http: www.emma-stil.de ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
A few points, from someone with a decent amount of testing background: * As someone who has contributed to OS projects, the lack of a pre-existing set of regression tests in org-mode is actually *the* most significant blocker to my getting involved in org-mode development. I do all of my coding - both professional and personal - in the context of tests, and not having the existing framework means that before I can start working on new features I first need to spend time yak-shaving on testing infrastructure. * I think there's a lot of over-thinking going on here. Here's the test-first coding discipline in a nutshell: 1. Identify a problem/missing feature. 2. Write a test (possibly using a unit-testing framework to help) which will pass when the bug has been fixed or feature added. This can be as simple as calling a function and validating its return value. 3. Run the test. Verify it FAILS. 4. Write code to make the test PASS. 5. Refactor, if you introduced any code duplication in step 4. 6. Run all the tests, to make sure you didn't break anything else. 7. Commit. If someone would be so kind as to identify a small bug or feature, I would be happy to demonstrate this workflow in the form of code, time permitting. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
(cc'ing the list again) Ben Alexander [EMAIL PROTECTED] writes: On 2008-Oct-23, at 17:26, Sebastian Rose wrote: Ben, I added this to the new page on worg. Please tell me: is it OK for you, if I simply add your ideas there?? Sure put what you want there. Great. FWIW, I'm brand new (today!) to git and I've cloned Worg today too. But I not a worger' -- I'm not sure I've read much of it online. So I'll look what else should be added? Hm - this thread is so busy, that I'd say I take a few hours on weekend to sum it up. What do the others think? Ben Alexander [EMAIL PROTECTED] writes: way too much Please, if you find time, add the rest somewhere there? If not, I'll take a closer look tonight and check in what's interesting (which seems to be all of it, actually). http://www.emacswiki.org/emacs-fr/UnitTesting: Does this mean to add code to Org's *.el files? This would mean a lot of work for Carsten :-( OK, once done... Uh, I hope that's not necessary. I'd hope that the tests could be written completely independently of any org-code. If the basic idea is to launch emacs in batch mode, then the test could redefine any function locally, in the test code, instead of screwing around with what Carsten has. If every test is written this way, it'll be very slow to run all the tests. Maybe that doesn't matter for bug-report tests? But here's where I should shut up and listen instead of talk. Great! I hope so too!! If slow, it's still no problem to run all those tests before a release or once a day, whatever. Glad to here your interested in keeping impact on Carsten low! Cheers, -- Sebastian Rose, EMMA STIL - mediendesign, Niemeyerstr.6, 30449 Hannover Tel.: +49 (0)511 - 36 58 472 Fax: +49 (0)1805 - 233633 - 11044 mobil: +49 (0)173 - 83 93 417 Email: [EMAIL PROTECTED], [EMAIL PROTECTED] Http: www.emma-stil.de ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
Re: [Orgmode] How you can help
On Thu, Oct 23, 2008 at 1:33 PM, Sebastian Rose [EMAIL PROTECTED] wrote: Hm - how about: 1. A not yet existent elisp file test-worg.el, that defines a function hello-worg, and a variable lang, and simply puts Hello Worginto the minibuffer (if (string= lang en)) and Hallo Worg (if (string= lang de)) ? I'd prefer something that would be of concrete value to someone in using org-mode - made-up scenarios tend to lead to unrealistic tests. Did you work with unit-testing frameworks for elisp already? Which one? I have not. I've done test-driven development in Java, C#, Ruby, and C++(!) though, and if I can do it in C++ I think I can get up to speed in any language pretty quickly ;-) Recommendations? As I said in another comment, I'm liking the look of Emacs Lisp Expectations a *lot*. Partly because I think Jay Fields is bang on the money for how maintain good habits in your tests, and partly because it has a mocking framework to go with it. Testing without a mocking framework is less fun than testing with. However, I haven't actually used it yet, so I can't responsibly recommend it. Let's just say that the discovery of it recently was the thing that got me excited about doing some serious ELisp development. Could say something about the effort to get started with such a framework? I think if you have some *running* examples to crib from it shouldn't take someone who is already ELisp-savvy more than a few hours to get the hang of it. Can we add it without changing Org's code? If understand 2. correctly - yes? In a dynamic language like ELisp, any testing framework that requires you to change the code just to get it under test isn't worth using. No, you should not have to change org-mode's code to get started with tests. That said, developing in concert with a test suite will definitely tend to influence how you write the code. Generally speaking, developing with tests encourages smaller functions which do one thing and one thing alone and don't have a lot of non-obvious inputs and outputs. Since this is usually the direction you *want* your code to go in anyway, it's a virtuous cycle. -- Avdi Home: http://avdi.org Developer Blog: http://avdi.org/devblog/ Twitter: http://twitter.com/avdi Journal: http://avdi.livejournal.com ___ Emacs-orgmode mailing list Remember: use `Reply All' to send replies to the list. Emacs-orgmode@gnu.org http://lists.gnu.org/mailman/listinfo/emacs-orgmode
