[Bug 1319397] Re: checkbox does not recover well from a suspend crash
[Expired for checkbox (Ubuntu) because there has been no activity for 60 days.] ** Changed in: checkbox (Ubuntu) Status: Incomplete = Expired -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1319397 Title: checkbox does not recover well from a suspend crash To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/checkbox/+bug/1319397/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1319397] Re: checkbox does not recover well from a suspend crash
Hey. Thanks for reporting this problem. Please look at my answers below: - Before running the test, I expected some sort of prompt, that would allow me to skip the test. Suspend/hibernate tests have a high likelyhood of failing, so I think this should be required. That's an interesting idea. It is not implemented right now. We have a similar bug that explores this idea already https://bugs.launchpad.net/plainbox/+bug/1232029 - Checkbox should save how far I got along in the tests more accurately. We do that. You should have seen a prompt if you want to 'resume', 'continue' or 'restart' (those are quite confusing) did you see that after restarting checkbox? - I'm not sure if this is the case or not, but the results should be uploaded before I reach the final tests. I am now dead set against running Checkbox, because I don't want to have to reboot my computer again. Ubuntu should be collecting stats on this, on how many people do not finish the tests and why. I don't think we need to upload results before every dangerous test. We try very hard to save everything on the filesystem before running each job so that if it crashes you should still be safe. Given the second question I'm setting this to incomplete until you reply if you saw the resume screen or not ** Changed in: checkbox (Ubuntu) Status: New = Incomplete -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1319397 Title: checkbox does not recover well from a suspend crash To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/checkbox/+bug/1319397/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
[Bug 1319397] Re: checkbox does not recover well from a suspend crash
Thanks for your prompt response! That's an interesting idea. It is not implemented right now. We have a similar bug that explores this idea already https://bugs.launchpad.net/plainbox/+bug/1232029 That bug report looks good. Is plainbox the backend for checkbox? I'm unfamiliar with it. We do that. You should have seen a prompt if you want to 'resume', 'continue' or 'restart' (those are quite confusing) did you see that after restarting checkbox? Yes, I did see that box. I clicked continue, as I deliberately wanted to skip the suspend tests. It started a microphone test, if I recall correctly, and then ran the suspend tests again, causing my computer to crash. I'll upload a log later tonight if I get other things off my plate first. The bug isn't that the dialog failed to appear, the bug is that it failed to skip a test that had previously caused my computer to crash, even though I chose continue instead of rerun last test. I don't think we need to upload results before every dangerous test. We try very hard to save everything on the filesystem before running each job so that if it crashes you should still be safe. Yes, but how do you know how many users never even try to run Checkbox again? Just because the data is saved on disk doesn't mean it will get uploaded. How do you know how many users give up before completing the process? My intuition says that a significant proportion do. There's a vast amount of UX materiel out there that examines form completion rates on the web. On the web, this stuff is measured, and you would be surprised how even tiny things put people off from completing a form. My proposal is simply to start measuring this by uploading every result as you go along, rather than assuming that the majority of users finish the process. Given that users are more likely to finish the process if nothing crashes, the fact that data isn't being uploaded in increments means that the received data is skewed towards test passes. Thinking about, this really is two bugs in one report, so let me know if I should file each one separately. -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1319397 Title: checkbox does not recover well from a suspend crash To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/checkbox/+bug/1319397/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
Re: [Bug 1319397] Re: checkbox does not recover well from a suspend crash
On Wed, May 14, 2014 at 6:14 PM, David D Lowe 1319...@bugs.launchpad.netwrote: Thanks for your prompt response! That's an interesting idea. It is not implemented right now. We have a similar bug that explores this idea already https://bugs.launchpad.net/plainbox/+bug/1232029 That bug report looks good. Is plainbox the backend for checkbox? I'm unfamiliar with it. Yes PlainBox is also a standalone tool but it is more low-level and aimed at generic testing rather than certification. It is also a development tool for test authors. Have a look at http://checkbox.readthedocs.org/en/latest/stack.html for details. We do that. You should have seen a prompt if you want to 'resume', 'continue' or 'restart' (those are quite confusing) did you see that after restarting checkbox? Yes, I did see that box. I clicked continue, as I deliberately wanted to skip the suspend tests. It started a microphone test, if I recall correctly, and then ran the suspend tests again, causing my computer to IIRC there are many instances of suspend tests. Do you remember which whitelist (test plan) did you select? If it really re-ran the suspend test that is a new bug (though we heard about that case happening from time to time but didn't manage to reproduce it). The gui is pretty buggy so if you can try the same set of test with plainbox that would help to nail the problem better (you want plainbox run -w /path/to/whitelist, make sure to run --help to decide what to do with results up front) crash. I'll upload a log later tonight if I get other things off my plate first. The bug isn't that the dialog failed to appear, the bug is that it failed to skip a test that had previously caused my computer to crash, even though I chose continue instead of rerun last test. I understand. I don't think we need to upload results before every dangerous test. We try very hard to save everything on the filesystem before running each job so that if it crashes you should still be safe. Yes, but how do you know how many users never even try to run Checkbox again? Just because the data is saved on disk doesn't mean it will get uploaded. How do you know how many users give up before completing the process? My intuition says that a significant proportion do. There's a vast amount of UX materiel out there that examines form completion rates on the web. On the web, this stuff is measured, and you would be surprised how even tiny things put people off from completing a form. We don't measure that and I cannot estimate how many users will do that. Uploading results (where?) before every test is a total overkill though. The current checkbox user interface (checkbox-gui) has lots of design issues but this is not a solution IMHO. My proposal is simply to start measuring this by uploading every result as you go along, rather than assuming that the majority of users finish the process. This is not possible at this time. We don't have anyone that could be dedicated to do such research, let alone maintain the infrastructure and coding changes. Still. If you want to contribute (maybe not by hacking on that exactly but by trying to find a solution together with us, please join us in the #checkbox channel on freenode and I'm sure we can figure something out as we go. Given that users are more likely to finish the process if nothing crashes, the fact that data isn't being uploaded in increments means that the received data is skewed towards test passes. We are not making any use of the data uploaded by users of Ubuntu AFAIK. That feature was related to the Ubuntu Friendly effort but it was discontinued. Checkbox is now mainly used by Canonical for professional services related to hardware certification. That doesn't mean the use case you are proposing is invalid. It simply means that's not what we can commit resources to. If you want to help, we'd be more than happy to see your contributions. Thinking about, this really is two bugs in one report, so let me know if I should file each one separately. I think the failure to skip is a very interesting bug and that's something I'm willing to investigate more. The other bug about what to throughout testing is a big open discussion. Feel free to open a bug if you are interested in re-designing checkbox to make it better for testers. Thanks for your contribution ZK -- You received this bug notification because you are a member of Ubuntu Bugs, which is subscribed to Ubuntu. https://bugs.launchpad.net/bugs/1319397 Title: checkbox does not recover well from a suspend crash To manage notifications about this bug go to: https://bugs.launchpad.net/ubuntu/+source/checkbox/+bug/1319397/+subscriptions -- ubuntu-bugs mailing list ubuntu-bugs@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-bugs
