Bram Moolenaar wrote: > So far we have been testing by hoping for people to install Vim and use > it. This does find problems related to daily use, but I suspect quite a > few things are not yet used and bugs go unnoticed. Then when Vim 7.3 is > released new features get used and bugs are uncovered. Too late!
A couple of points: 1/ Human tests are certainly complementary with automated tests. Having said that, I'm mostly in favor of adding more automated tests (make test) or improving coverage of existing ones. I've never looked at how to write those tests myself, but I can give it a try. 2/ One thing that surprised me in Vim source code at first (but I got used to it) is the lack of asserts. Why not put asserts in Vim code? I think they could catch problems during development and they can be disabled for official builds. Personally, I would build my own Vim with asserts enabled to be able to find and report more bugs. They also add semantic information when reading the code and can even help to improve accuracy of Coverity checkers. 3/ Speaking of Coverity, the Coverity scan page shows that there are 25 uninspected Vim defects (http://scan.coverity.com/rung2.html). I don't know how up-to-date this is and I don't have access to it, but would it be possible to look at them before releasing Vim-7.3? 4/ Some bugs can also be found if we test Vim on different OS. How about keeping track of whether Vim successfully compiles and passes all tests on: - Linux (x86, x86_64, ARM, MIPS, ... ) - Windows 95, 98, NT, 2000, XP, Vista, 7 (32 & 64 bits) - Cygwin-1.5, Cygwin-1.7 - OS-X - OpenBSD, HP-UX, Solaris, AIX... - and the more exotic zOS, vms, qnx, AmigaOS, BeOS, RISC-OS (others?) We can ask users to submit short summaries like this and collect them somewhere: Vim version: Vim-7.3c (2427:33148c37f3c9) OS: Ubuntu-10.04, x86 configure: --with-features=huge --enable-gui=gnome2 compiler: gcc-4.4.3 compilation: OK, no warnings. make test: ALL DONE Tested by: Dominique Pelle File vim/src/Makefile contains this kind of information but it looks old. 5/ Many bugs have also been found thanks to Valgrind. I use it often to check Vim but I'm sure that I did not think of every configurations and odd cases. I wrote a help page in case others want to also test Vim with Valgrind: http://dominique.pelle.free.fr/valgrind.txt.html Latest version of Valgrind in SVN is supposed to run on OS-X and Linux ARM (I don't have access to those OS). Maybe some developers have access to other runtime checkers or static analyzers? > I have started a shared document for this. it's very rudimentary: > https://docs.google.com/document/edit?id=1CyOvJqKYoTmL4D50w2htYMG7xt5SADcIX6r7zZ978HQ&hl=en&authkey=COfQ_6oD What? Not a document that I can edit with Vim? :-) Regards -- Dominique -- You received this message from the "vim_dev" maillist. Do not top-post! Type your reply below the text you are replying to. For more information, visit http://www.vim.org/maillist.php
