|
I'd like to see a log that make it obvious what the error was and which
test caused it. If the current tinderbox mechanism can't do that -- because of bugs, limitations, features, etc. as described below -- perhaps we need to think about how to change the way tinderbox works so that it can correctly report errors and make them obvious. John Heikki Toivonen wrote: The issue of Tinderbox log readability has been raised (once again). The purpose of this email is to start a discussion to see if we can come to a consensus on a suitable format. (We are talking about the full Tbox log.)To start with, let me describe briefly how the current system works. A Tinderbox clients pulls chandler source code, do build if code changed, then run various tests. All of this activity is logged, and of course Chandler itself and the test frameworks provide additional log files. At the end of the run, the client sends an email to the Tinderbox server with all of the logs concatenated to a single file. In some cases we can miss entire logs from certain tests if we encountered a Python crash. For example, functional tests will not output anything until they finish, and if Python crashes before we finish there will be no output. The only indication in the log file is that there is a line that says something like 'starting functional tests' and the next line says exit code=<some non-zero number). We can also experience a situation where test logs were output properly and all tests passed, but then we experience a Python shutdown crash. We catch that in some cases (but not all), and this will show up in the Tbox log as something like 'all tests passed' followed by line that says 'tests failed' (the first one was output by the test framework, the second one by the script that started the testsuite and monitored python exit value). The Tinderbox server creates an HTML page from the log. The page contains the full log, among with some additional information. At the top of the page is a section of links to errors. That is produced by parsing the full log, and creating links for each line were certain keywords appear, like 'error', 'warning' and 'failed'. This is of course not foolproof, and can result in false positive links. The longer the log, the more false positive error links there are the top. Currently the logs are very long and verbose. Now... What should we have in the logs in your opinion, and how should they be formatted? |
_ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ _ Open Source Applications Foundation "chandler-dev" mailing list http://lists.osafoundation.org/mailman/listinfo/chandler-dev
