Any opinions on this? I fear the mail drowned in other mails on the list. -- .marius
On May 1, 2012, at 3:33 PM, <[email protected]> <[email protected]> wrote: > Hi, > > Back in the days, when we were approaching a new release of Qt, we would use > a black team which would be responsible for driving the testing of a new > package. Their main focus would be to test every new package generated, both > source and binary packages, and try to break them with the most used > configuration (every configuration is not possible due to the vast number of > configurable options, and system configurations). > > When doing this they would have a common list of issues to look out for, to > make sure that the reporting had some structure (same report fields for each > reporter/platform/package) and that they wouldn't skip something for one > package, etc. The list contained things like: > General: > Report date > Testers name > Package name > Package type > Build date > Makespec used > Source packages: > Configuration line for main testing > Configure asks about license when run with no license options > Compiling the package > Compiles with minimal options (e.g. -opensource -confirm-license) > Compiles as a static build, where supported > Compiles in namespace, where supported > Compiles with shadow build > Cross-compiles, where supported > Installing the package works > "make install" to system directory as root works > "make install" to local prefix as regular user works > "make install" distributes files correctly on Mac > "make clean" and "make distclean" works > Text files have the correct EOL for the type of package? > Files/directories in the package have sane file permissions and timestamps? > Tags (%VERSION%, %SHORTVERSION%) have been replaced properly > README has valid information about how to compile the package on the platform > tested > Binary packages: > Fresh install > Installer is correctly signed, e.g. Windows shows correct vendor/certificate > data and no warnings about untrusted vendor. > Installer displays appropriate graphics, strings, version numbers > Installer offers the correct license(s) > Installer offers sane default install directory (including version number) > Installer correctly installs to default directory > installer correctly installs to non-default drive/directory > Installer sanely reports progress and completion > Installer installs only selected components > Component selection works sanely > Shortcuts from last page of installer work (e.g. shortcuts to README, demos, > etc) > Installer correctly creates desktop shortcuts and they all work > Installer correctly creates Start Menu shortcuts and they all work > Environment settings are correct in Qt MSVC Command Prompt > Package shows up in Control Panel/Package Manager with correct > description/version > Any patching of files has been done correctly, e.g. patching of library paths > into .exe files and the paths hard-coded in the QtCore library. > Upgrade install, if supported > - As above > Parallel install, if supported > - As above > - Can't install over the top of an existing package without being warned > - Shortcuts point to the right package > - Previously installed packages still work > Uninstall > - Removes all installed files, > - Removes all empty dirs, > - Removes registry keys, > - Reverses any other changes made by installer > - What to do with data files created by installed files, e.g. saved > settings and files created by demo apps? > - What to do with data shared between more than one package instance? > Aborting installation, if supported > - Cancel button is available > - Installer cleans up, as for full uninstallation > Failed installation > - insufficient disk space (can be faked by installing onto a small USB > key/SD card) > - network problems while using online installer > Both package types: > Verify the license > Assistant works > Designer works > "Showcase" demo/example apps launch without crashing > "Showcase" demo/example apps function acceptably. > Demo/example apps can be rebuilt > External Qt apps can be built against the package (e.g. Qt Creator) > Did "DLL Swapping" on a Qt app (e.g. KDE, Google Earth, Qt Creator) work? > Click around various examples/demos for a while (GUI stress-testing) > Audio/Video w/phonon > Audio/Video w/QtMultimedia > Raster paint engine works > Image formats work > Graphicsview > OpenGL > OpenVG > Printing > QML > QtNetworking > QtScript > QtSql > QtSvg > WebKit works? > Logging into a popular site (facebook, hotmail, gmail etc) using the > demo-browser > Uploading an image to imgur.com using the demo-browser > Watch a cat video on youtube using the demo-browser > Xml > qmlviewer (outside of qtdemo) > Thanks Jason for a thorough list from the Qt 4 days! > This testing should be distributed on as many people as possible to lower the > load on individuals, and to allow us to get as much feedback as quickly as > possible. This in turn allows for shorter test periods, and that way we can > get the Qt releases out according to our schedule(s). > > The question is now, how do we organize this? We need a form, and a place to > collect them and allowing us to easily browse the reports and see the status > for any given release. > > I don't think using SurveyMonkey would cut it with this scope. :) > > How about maybe installing the "JIRA Issue Collector" plugin (You are not > required to have a Jira account to use the issue collector, as it can use a > fall-back account for general reporting!), allowing us to report against a > "Fix Version", and perhaps use "Labels" to distinguish between build dates? > Maybe these reports should have their own types (currently we only have > Bug/Task), like "Test Report"? Then on a no-go report, Jira "Bug" issues > could be created and linked to the "Test Report"? > > Opinions? > -- > .marius > > _______________________________________________ > Development mailing list > [email protected] > http://lists.qt-project.org/mailman/listinfo/development
_______________________________________________ Development mailing list [email protected] http://lists.qt-project.org/mailman/listinfo/development
