Hi Johannes,
> -----Original Message----- > From: Johannes Schlüter [mailto:johan...@schlueters.de] > > [...] > > The PFTT Unattended Test System I have running sends only a summary > > report to limit the amount of email messages. > > I think sharing this is good. My peeve is that these summaries are > *looooooong*. Taking the length and amount makes it hard to digest them. > What I'd like are three things which should also assist casual > contributors: > > A. Instructions to reproduce the tests somewhere on qa.php.net or > such, so that when an issue is found I (or anybody else) can > reproduce it even their primary development system uses a > different setup Yes, perhaps we should create a howto or similar on how to read and reproduce an issue found from one of these reports. We have some documentation on PFTT here: https://github.com/OSTC/PFTT2/blob/master/README.mdown. In many cases if there is a test failure it is not necessary to specifically use PFTT to reproduce the failure, but key information such as the PHP configuration or the diff of the actual and expected results can only be found in the result packs (which are linked to in the summary reports). > B. A short summary where I can see, with a single look, whether > there's something bad or not. With the current 78K I can scroll > over it and still now as much as before. The results we would *like* to send are in HTML format and look more like this: http://windows.php.net/downloads/snaps/ostc/pftt/PHP_5_5/rc7ee97e/PHPT_CMP_PHP_5_5-r5e5ecae-NTS-X86-VC11_Opcache_Local-FileSystem_MySQL-5.6_CLI_WinCacheU_v_PHP_5_5-rc.html This is not a summary report, but a full report for just a single scenario - in this case comparing two NTS builds using CLI, with Opcache+Wincache enabled and with tests running on the local file system (we also test SMB, DFS and deduplication scenarios). You can see from the report above the summary includes easy to read columns with +n or -n diff for things like test failures between the two builds. Generally when we look at this we can immediately see that there are new test failures, find which test(s) failed and then dig into the result pack for detailed debugging data. > C. have a database with a nice report to get the details. Is there a database currently available or any ideas on how to get this going? > > This wish is not limited to your tests, but also the "normal" tests we already > have, there https://qa.php.net/reports/ is a good start for reporting, but > still > so much room for improvement (i.e. trend graphs or "alerts" or something) > > And to be absolutely clear: I thin it is *great* you're testing this, and > *great* > you are sharing the results, but I guess currently the results reach some void > and we should improve that. > Thanks for the feedback. We definitely want to continue iterating on this and improve the reports, and hopefully get folks looking at this data rather than tuning it out. Thanks, Steve
