> On Mar 20, 2017, at 2:34 PM, Kent Fredric <kentfred...@gmail.com> wrote: > > On 21 March 2017 at 08:02, Doug Bell <madcity...@gmail.com> wrote: >> * "X" is the main type >> * PASS = 1 >> * FAIL = 2 >> * NA = 3 >> * UNKNOWN = 4 >> * "Y" is the step of the process we made it to >> * 0 is "complete/unknown" >> * 1 is "extract" >> * 2 is "build" >> * 3 is "test" >> * 4 is "install" >> * "Z" is a step-specific status code >> * "0" is always "generic/unknown" >> * But for the "build" step >> * 1 can mean "OS unsupported" >> * 2 can mean "Compiler needed" >> * 3 can mean "Dependencies failed" >> >> This might be over-engineering > > > I get the impression there might be scope for >1 value of Y and Z in a > response. > > For instance, the flow assumes each stage blocks the next .... > > However, as far as assumptions go, we also assume that people won't > run tests if dependencies are missing, > and that assumption is proving to be wrong. > > So for instance, what if: > > 1. Configure fails, but then the user interactively fixes it and continues > 2. Test fails, but the user installs anyway > 3. Install fails > > All these 3 things could happen in the scope of a single report. > > </overengineering>
This problem is the same even without granular status codes: Configure fails, user patches to fix it, tests succeed, install succeeds. Is it a PASS report, an NA report (currently used (I think) for every configure failure), or a FAIL report (the configure failure was erroneous, since it clearly now works on this platform)? How can the user interactively fix a configuration problem that would also then be considered a failure in configuration? If the configure script allows it, it must be normal operation, no? If it involves editing the config script, that's a different run (and an edited dist, which is not good report fodder). If you're force-installing, I'd say you send in the test failure. In theory, the "step" field could be defined as "the last step completed successfully", and the report just also has in it the details of a failed force-install (because at that point, there's no good way to say that the install would've failed on its own if the tests had not found any problems). Additionally in this same context of multiple status codes: One of the potential benefits could be detecting a reason for the tests to fail. It's easy to parse out "Module missing", for example, and come up with a status code to categorize that. But what if there were multiple reasons? I'd say that the "test" step is likely to not have more granular status codes unless the QA / TAP teams can come up with ways of expressing them, and only one applies (for everything else, there's analysis.cpantesters.org <http://analysis.cpantesters.org/>).
