Re: Default Desktop Experience for 11.04 - User testing results
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Jono Bacon wrote on 16/04/11 20:05: On Fri, 2011-04-15 at 03:00 +0100, Matthew Paul Thomas wrote: ... Last week, Charline Poirier ran a user test of Unity, with 11 individual participants. This week, I have helped Charline analyze the results. ... Wonderful work, and now very visible work: http://news.slashdot.org/story/11/04/16/0239213/5-Out-of-11-Crashed-Unity-In-Canonicals-Study ;-) Up to Slashdot's usual standards, I see. To correct the obvious from that post: * No, the results of the study have not been published. Charline will do that soon. * No, my name is not Rick Spencer. (rickspencer4?) * The object of the study was, obviously, not to measure crashes. Crashes are usually quick to find and fix, so any user test of those would be weeks out of date when published. I mentioned them only as a reminder that to users, bugs are indistinguishable from design flaws, and vice versa. (For example, one test participant pressed Ctrl Alt F1 apparently by accident, and ended up at a console. This wasn't a crash, but it had exactly the same effect as one.) I think this feedback points to a series of design and engineering bugs that we need to resolve in 11.10. Have the design bugs been filed in Launchpad? Charline has been working with John Lea on that today. I think it could be worthwhile to rate the prioritization of the design bugs based upon the level of success in your study. As an example, if 1/12 achieved a task, it would be a high priority bug, as opposed to if 10/12 achieved the task it would be a low priority bug. ... I'll pass that on to John. Cheers - -- mpt -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.11 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2tokcACgkQ6PUxNfU6ecr1TgCePApLBm/ySLUAasdjINBrWQHs LKYAoNLXk9mfWvVzz9fF7h4rl3eBq9u3 =j4k8 -END PGP SIGNATURE- -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
Matthew Paul Thomas schreef op di 19-04-2011 om 15:55 [+0100]: * The object of the study was, obviously, not to measure crashes. Crashes are usually quick to find and fix, so any user test of those would be weeks out of date when published. I mentioned them only as a reminder that to users, bugs are indistinguishable from design flaws, and vice versa. (For example, one test participant pressed Ctrl Alt F1 apparently by accident, and ended up at a console. This wasn't a crash, but it had exactly the same effect as one.) Maybe we need to add a line of text above the login prompt somehow, that tells the user what key to press to get back to their GUI? (This might be difficult to do correctly with multiple logins etc. though?) -- Jan Claeys -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On 19 April 2011 15:13, Jan Claeys li...@janc.be wrote: Matthew Paul Thomas schreef op di 19-04-2011 om 15:55 [+0100]: * The object of the study was, obviously, not to measure crashes. Crashes are usually quick to find and fix, so any user test of those would be weeks out of date when published. I mentioned them only as a reminder that to users, bugs are indistinguishable from design flaws, and vice versa. (For example, one test participant pressed Ctrl Alt F1 apparently by accident, and ended up at a console. This wasn't a crash, but it had exactly the same effect as one.) Maybe we need to add a line of text above the login prompt somehow, that tells the user what key to press to get back to their GUI? (This might be difficult to do correctly with multiple logins etc. though?) We could do like Fedora and have X on virtual terminal 1. Why do we need 6 virtual terminals anyway? Jeremy -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
because screen is hard mode. On 20 April 2011 05:07, Jeremy Bicha jer...@bicha.net wrote: On 19 April 2011 15:13, Jan Claeys li...@janc.be wrote: Matthew Paul Thomas schreef op di 19-04-2011 om 15:55 [+0100]: * The object of the study was, obviously, not to measure crashes. Crashes are usually quick to find and fix, so any user test of those would be weeks out of date when published. I mentioned them only as a reminder that to users, bugs are indistinguishable from design flaws, and vice versa. (For example, one test participant pressed Ctrl Alt F1 apparently by accident, and ended up at a console. This wasn't a crash, but it had exactly the same effect as one.) Maybe we need to add a line of text above the login prompt somehow, that tells the user what key to press to get back to their GUI? (This might be difficult to do correctly with multiple logins etc. though?) We could do like Fedora and have X on virtual terminal 1. Why do we need 6 virtual terminals anyway? Jeremy -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Missing panel items (was Re: Default Desktop Experience for 11.04 - User testing results)
Hello, * The weather notifier is missing. I really like this little notifier so I know when to throw open the windows and get some fresh air! Install the indicator-weather package - it's in the official Natty repos (universe). * System monitor. I like this little applet because it gives me a quick way to take the pulse of my system. How hard it's running, is there a lot of network activity going on, etc. For the CPU and memory usage you can use System Monitor Indicator [1] * Force quit. Some times you just need it and a kill -9 just isn't easy to get to. There's no appindicator for this, but you can create a .desktop file (just copy any from /usr/share/applications/ on your desktop and edit it as a template), and under Exec enter xkill, then drag and drop this .desktop file to the Unity launcher. Then to force an application to close, click this icon, then click the window you want to close. I hope this helps :) [1] http://www.webupd8.org/2011/03/system-monitor-indicator-puts-cpu-and.html -Andrew On Mon, Apr 18, 2011 at 16:02, Barry Warsaw ba...@ubuntu.com wrote: Now that I'm using Unity a lot, I find that I'm missing a few things from the classic desktop's upper panel. I don't know how to get these back, if it's even possible. * The weather notifier is missing. I really like this little notifier so I know when to throw open the windows and get some fresh air! * System monitor. I like this little applet because it gives me a quick way to take the pulse of my system. How hard it's running, is there a lot of network activity going on, etc. * Force quit. Some times you just need it and a kill -9 just isn't easy to get to. Cheers, -Barry -- ubuntu-devel mailing list ubuntu-de...@lists.ubuntu.com Modify settings or unsubscribe at: https://lists.ubuntu.com/mailman/listinfo/ubuntu-devel -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On Sun, 2011-04-17 at 09:17 +0100, Paul Sladen wrote: On Fri, 15 Apr 2011, Rick Spencer wrote: For brand new users? Some of the tasks aren't relevant. Which ones? Changing the background image and setting in general. On the contrary. Changing the wallpaper and screensaver are comforting personalisation steps that new users seem to undertake on their own. That was my point. You don't personalize something until you *own* it. Until you've used it enough that you know how you want to own it. It's not something you do while you are carrying out other tasks. In any case, it's a moot point, since changing the desktop background was something that the study found users could do easily, they were blocked by the usability engineer telling them not to do it in the obvious manner. Cheers, Rick -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On Fri, 2011-04-15 at 03:00 +0100, Matthew Paul Thomas wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rick Spencer wrote on 08/04/11 02:38: ... Back at UDS for 11.04 in Orlando, Mark set the goal of using Unity by default on the Ubutu desktop. Given the current course of development, it appears that we are going to achieve this goal, and Unity will stay the default for 11.04. I'm following up on this list at the suggestion of the Tech Board to give folks a chance to respond or escelate any concerns. ... Last week, Charline Poirier ran a user test of Unity, with 11 individual participants. This week, I have helped Charline analyze the results. snip Wonderful work, and now very visible work: http://news.slashdot.org/story/11/04/16/0239213/5-Out-of-11-Crashed-Unity-In-Canonicals-Study ;-) I think this feedback points to a series of design and engineering bugs that we need to resolve in 11.10. Have the design bugs been filed in Launchpad? I think it could be worthwhile to rate the prioritization of the design bugs based upon the level of success in your study. As an example, if 1/12 achieved a task, it would be a high priority bug, as opposed to if 10/12 achieved the task it would be a low priority bug. I just want to ensure that we build this feedback into the design and engineering iteration process. Jono -- Jono Bacon Ubuntu Community Manager jono(at)ubuntu(dot)com www.ubuntu.com : www.jonobacon.org www.twitter.com/jonobacon : www.identi.ca/jonobacon -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On Fri, Apr 15, 2011 at 1:48 AM, Bryce Harrington br...@canonical.com wrote: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. What would be telling is if the people who didn't figure out the task the first time remembered how to complete it the next time. That shows that the task is learnable, which is acceptable if the curve matches the difficulty of the functionality. Unfortunately very rarely do you use the same users for multiple tests (in fact it is usually discouraged unless you are doing longitudinal studies) except in longitudinal studies. I'm curious whether these stats are higher/lower/same-as with Classic Desktop. IOW this needs a control group so we can tell if the new design brings improvement or regression. Also, these tests measure usability, but not their overall impression. Did they like it? Find it frustrating/confusing? Do people in the UK use the System Usability Scale, NASA TLX, or Modified-Cooper Harper? Theyre assessment surveys that measure satisfaction and cognitive load, but I've only seen human factors engineers use them. Bryce -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop -- Celeste Lyn Paul KDE Usability Project KDE e.V. Board of Directors www.kde.org -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Bryce Harrington wrote on 15/04/11 06:48: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. I'm curious whether these stats are higher/lower/same-as with Classic Desktop. IOW this needs a control group so we can tell if the new design brings improvement or regression. Sure, that's the main problem with using this data. The test was intended to find fixable problems in Unity, not to compare Unity with Classic. And the tests Canonical has done on Ubuntu previously have mostly been on other tasks, such as importing photos and music and burning CDs. Also, these tests measure usability, but not their overall impression. Did they like it? Find it frustrating/confusing? ... In my limited experience, user test participants can often be highly complimentary about an interface that completely failed them (or critical of an interface where they had no trouble). They can also start guessing about what other people what other people might think, or guessing about how easily they might learn something later. That said, here's a representative sample. P1: I've got a Mac, and I think it's quite similar to a Mac, the layout, which made it easier to use. I think people going from Windows to this would need a little help, because it's a very different layout. And: It's a nice casual-ish font that you've used here. P2: I'm guessing the way it's set at the moment, with the wallpaper, the background, I'd personally want everything to be crisper and clearer and more obvious ... I find it all a bit blending into each other, with those settings. But I do like the overall design of it, with the nice curves and the nice icons at the side, and the font used, and the design of it's really nice. P3: It's prettier than a lot of current operating systems. And: I don't think it's a complicated operating system ... it would just take time. P4: Some of the features, like when you minimize something you don't find the menu bar, so it's quite hard to find. And: The simple is quite simple, not too hard to follow or too hard to understand. If possible, if you could give them a demo. This is where you go to get a file, this is where you get all your icons, for the first time user, obviously. P5: I really quite like it. I think it's quite intuitive, with the exception of the favorites, making an application a favorite, which obviously I wasn't able to achieve. I wouldn't be baffled about how to use this operating system for the first time, if I didn't have a manual to read ... It's quite clean-looking, it's quite modern-looking. It seems to me to be closer to a Mac-style operating system than to a Microsoft-style operating system. P7: No. I don't know. So at this point, I would have to say my initial impressions are -- I wouldn't write it off, because I've heard too many good things about it, but my initial impressions are 'Damn, I'm going to need a manual' ... I don't really want to do that, which is a bit lazy of me, I know. At the end: It's not really working for me. I'm finding it time-consuming and slightly confusing at times ... That's my kind of bottom-line impression, really, that the differential [between Windows and Ubuntu], the gap is really very big. P8: Maybe the settings should have been more prominent, but I found them eventually. Fairly standard. If it can run anything I wanted, I would consider it. P9: It's okay ... It's not as confusing as the Mac. P10: So generally I think it's pretty good. I think it needs some work on it. I think making it a bit more intuitive. P11: Oh yikes, is it like Apple? ... I never liked the [Mac] interface. I didn't know where I stored things, where I put things. I like these pictures. It's aesthetically pleasing. It also looks like it's child-friendly, because it's pictures. And it looks more ordered than Apple. So I want to play with it now. At the end: I think it's pretty. It's aesthetically pleasing. P12: And I really like the cleanness of it all. It's tidy. At the end: It's quick and responsive. I'm still frustrated that I'm not able to find anything like My Computer. And hardware as well, I want to see what the hardware is, the versions of the graphics and drivers. - -- mpt -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/
Re: Default Desktop Experience for 11.04 - User testing results
First, thanks Usability Team! I know how much work goes into planning and running a study like this, and how much agony is involved in interpreting and writing up the results. It's clear that there are some areas for improvement in 11.10, and these results will be instrumental in helping to guide those investments. On Thu, 2011-04-14 at 22:48 -0700, Bryce Harrington wrote: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. Well, think back to the last time you got a new device. For example, if you have an Android phone. You are probably pretty facile with the interface now, but if someone handed it to you and said do this task with it you may have struggled to some basic things, like launching apps. A lot of the fun for users in getting a new devices is learning how to use it. Given my experience, these numbers look they could be improved, but I don't find them particularly concerning. For example, you can easily use Unity quite productively before you learn that you can add items to the launcher, or change the background picture, though I suspect many Windows users would right-click and the desktop to set the background picture and would do fine. The only area here that is at all concerning to me regards launching applications. I'd like to see some focus on the application lens in 11.10 (see what I did I did there? :) ) I'm curious whether these stats are higher/lower/same-as with Classic Desktop. IOW this needs a control group so we can tell if the new design brings improvement or regression. For brand new users? Some of the tasks aren't relevant. For others, like finding installed applications, I presume this was dead easy in Classic GNOME, but it's hard to say for a certain. Also, these tests measure usability, but not their overall impression. Did they like it? Find it frustrating/confusing? This is actually a very important question. For me, when I go back to Classic, it feels very old and mundane. Many theories hold that the aesthetics trump usability, or at least strongly influence the perception of the usability of a system. In other words, given 2 identically design systems that only differ in terms of theming, for example, users will rate the system with the more pleasing design to be more usable and are more likely to start using it. Cheers, Rick -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
2011/4/15 Matthew Paul Thomas m...@canonical.com: * 8/10 people could find a window's menus, but 7/8 of them learned to access them by hovering over maximized close/minimize/unmaximize buttons then moving horizontally -- which was extremely slow, and failed whenever the window wasn't maximized. Why not swap the default state of the panel from one where the application name is shown in full and the menus hidden, to the alternative where the menus are shown, and long application names are partially obscured? Then, allow that if someone mouse-overs the application name, it reveals itself fully; in this case, an arrow symbol (for example) could be included to make this action discoverable. I believe this swap wouldn't cause any usability problems (unless knowing the long name of a window is critically important), but instead would make the location of window menus obvious. Of course, if it is intentional that menus are hard to discover (to push application developers towards Chrome-like streamlining), or if there would be problems due to claims of Mac-mimicry, then it clearly makes sense to keep the current behaviour. -- Toby Smithe http://launchpad.net/~tsmithe -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On Fri, Apr 15, 2011 at 07:08:02AM -0700, Rick Spencer wrote: On Thu, 2011-04-14 at 22:48 -0700, Bryce Harrington wrote: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. Well, think back to the last time you got a new device. No, I definitely agree there, but this is why I suggested a control group - to help understand if it's just ordinary newbie level learning curve stuff. Like, maybe with Classic Desktop only 2/10 new users could find a new game (esp. if they're equating USC with the recycling bin...) so 6/10 with unity would be a huge step forward. On the other hand if with Classic it's at 9/10 then I think you'd be right that it's an area needing further attention. In any case, I'm loving the numbers, you know how much I geek out on quantification of issues. :-) I would love to see this same test repeated again every so often, so we can chart the results and use them to demonstrate that we're getting better. mpt's additional info adds a lot more context to the numbers and makes them sound a lot less troublesome, but I'll comment against that one separately. Bryce -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On Fri, Apr 15, 2011 at 01:33:27PM +0100, Matthew Paul Thomas wrote: Bryce Harrington wrote on 15/04/11 06:48: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: Also, these tests measure usability, but not their overall impression. Did they like it? Find it frustrating/confusing? ... In my limited experience, user test participants can often be highly complimentary about an interface that completely failed them (or critical of an interface where they had no trouble). They can also start guessing about what other people what other people might think, or guessing about how easily they might learn something later. Good point. Even so I find the comments on their impressions adds a lot of context to the numbers, so thanks muchly for forwarding them. AIUI, ensuring a good initial impression was one of the major goals for the new design work, and judging from the comments this was definitely achieved. Most everyone had positive comments as to the aesthetics. My favorite is: So I want to play with it now. Of the negative comments, the theme seems to be around learnability, which may also be a quirk of the testing process (people may have felt pressure to do the given tasks quickly so they can complete the test). But this issue has also gotten a fair bit of airing on blogs and mailing lists. I've heard several sources also make P4's suggestion about making demo videos easily available. Jorge put out a demo video on how to multitask in Unity, which I found very engaging and enlightening. Ideally stuff should be discoverable, but until it is maybe an effective workaround would be to just ensure demo videos are easily at hand? Like a hyperlink to an online collection of videos that's provided in the global help menu, on the desktop, or on the welcome page of the webbrowser or something. Bryce -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rick Spencer wrote on 15/04/11 15:08: First, thanks Usability Team! I know how much work goes into planning and running a study like this, and how much agony is involved in interpreting and writing up the results. It's clear that there are some areas for improvement in 11.10, and these results will be instrumental in helping to guide those investments. Charline did all the planning and test moderation. I was just the stenographer afterwards. Later on, Charline will publish a full report on the test. I just wanted to post a quick summary in time to be helpful for the default experience discussion. On Thu, 2011-04-14 at 22:48 -0700, Bryce Harrington wrote: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. Well, think back to the last time you got a new device. For example, if you have an Android phone. You are probably pretty facile with the interface now, but if someone handed it to you and said do this task with it you may have struggled to some basic things, like launching apps. A lot of the fun for users in getting a new devices is learning how to use it. I didn't have anything close to that kind of trouble when trying out an Android phone. (Though like anyone on a developer mailing list, I'm not a representative sample.) ... For brand new users? Some of the tasks aren't relevant. Which ones? ... Also, these tests measure usability, but not their overall impression. Did they like it? Find it frustrating/confusing? This is actually a very important question. For me, when I go back to Classic, it feels very old and mundane. Many theories hold that the aesthetics trump usability, or at least strongly influence the perception of the usability of a system. In other words, given 2 identically design systems that only differ in terms of theming, for example, users will rate the system with the more pleasing design to be more usable and are more likely to start using it. ... This is the aesthetic usability effect. http://usabilityfriction.com/2010/03/30/aesthetic-usability-effect/ http://jnd.org/dn.mss/emotion_design_attractive_things_work_better.html - -- mpt -BEGIN PGP SIGNATURE- Version: GnuPG v1.4.10 (GNU/Linux) Comment: Using GnuPG with Mozilla - http://enigmail.mozdev.org/ iEYEARECAAYFAk2oxq0ACgkQ6PUxNfU6ecoIcACcCcdaRX0LStM999wHOWz+q/GU 4YEAn05ZbWUmv1dPOkBJzHvpYb8eSaXg =I/jZ -END PGP SIGNATURE- -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
On Fri, 2011-04-15 at 23:29 +0100, Matthew Paul Thomas wrote: -BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rick Spencer wrote on 15/04/11 15:08: First, thanks Usability Team! I know how much work goes into planning and running a study like this, and how much agony is involved in interpreting and writing up the results. It's clear that there are some areas for improvement in 11.10, and these results will be instrumental in helping to guide those investments. Charline did all the planning and test moderation. I was just the stenographer afterwards. Later on, Charline will publish a full report on the test. I just wanted to post a quick summary in time to be helpful for the default experience discussion. On Thu, 2011-04-14 at 22:48 -0700, Bryce Harrington wrote: On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. Well, think back to the last time you got a new device. For example, if you have an Android phone. You are probably pretty facile with the interface now, but if someone handed it to you and said do this task with it you may have struggled to some basic things, like launching apps. A lot of the fun for users in getting a new devices is learning how to use it. I didn't have anything close to that kind of trouble when trying out an Android phone. (Though like anyone on a developer mailing list, I'm not a representative sample.) Oh? I recall learning many things on my phone. How to launch apps. How to use the task manager to switch between running apps. How to enter text in general and swype in particular. How to pull down the notification area and how to push it up again, etc... If you were watching over my shoulder you may have said Fail Fail Fail many times. Perhaps you would have been correct, the usabliity could be better. However, it's easy to forget the first few times you satisficed through something compared to the hundreds of times you've done something after you mastered it. ... For brand new users? Some of the tasks aren't relevant. Which ones? I believe the following are tasks indicative of repeated and experienced usage, not first time usage: * Changing the background image and setting in general. * Adding a game to the launcher. * Rearranging items in the launcher. And these were some that users had the most trouble with. Cheers, Rick -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
Re: Default Desktop Experience for 11.04 - User testing results
-BEGIN PGP SIGNED MESSAGE- Hash: SHA1 Rick Spencer wrote on 08/04/11 02:38: ... Back at UDS for 11.04 in Orlando, Mark set the goal of using Unity by default on the Ubutu desktop. Given the current course of development, it appears that we are going to achieve this goal, and Unity will stay the default for 11.04. I'm following up on this list at the suggestion of the Tech Board to give folks a chance to respond or escelate any concerns. ... Last week, Charline Poirier ran a user test of Unity, with 11 individual participants. This week, I have helped Charline analyze the results. In this summary, I have numbered the participants: - - P1, 19, a student and Mac user - - P2, 33, an administrator and Mac user - - P3, 25, a student and Windows user - - P4, 32, a teacher and Windows user - - P5, 27, a compliance officer who uses both Windows and Mac - - P7, 44, a life coach who uses both Windows and Ubuntu - - P8, 30, an IT network manager and Windows user - - P9, 22, a student and Windows user - - P10, 21, a student and Windows user - - P11, 47, a teacher and Windows user - - P12, 34, an operations manager and Windows user. The test machine was a Lenovo ThinkPad T410i running Ubuntu Natty with unity 3.8.2-0ubuntu1 and compiz 1:0.9.4git20110322-0ubuntu5. Charline asked each participant to try several tasks. Not every participant had time to try every task. * Every participant who was asked understood most of the launcher items. P7 and P11 thought that LibreOffice Calc was a calculator, and P7 and P9 thought Ubuntu Software Center was the Recycle Bin. Nobody understood Ubuntu One. (The Classic session has much smaller icons for everything, but has a visible-by-default label plus an extra tooltip for each app.) * Almost everyone understood most of the indicators, but 4/11 people (P7, P9, P11, P12) thought the Me menu icon might be a close button. * 11/11 people easily launched Firefox to check Web mail. * 8/10 people could find a window's menus, but 7/8 of them learned to access them by hovering over maximized close/minimize/unmaximize buttons then moving horizontally -- which was extremely slow, and failed whenever the window wasn't maximized. * 10/11 worked out how to open a new Firefox window, though 5/10 first tried clicking Firefox in the launcher again, which didn't work. * Only 4/11 worked out how to change the background picture. This is not as bad as it looks: for some of the others, Charline had asked them *not* to right-click on the desktop, because she was testing access to settings in general. Nevertheless, no-one found System Settings, in the session menu or anywhere else. * Only 5/11 could easily rearrange items in the launcher. For the other six, the main problems were that the launcher scrolled when they were trying to drag an item, or that it didn't accept a drop. (P10 was particularly unlucky in doing a Dash search for menu and finding the Main Menu editor, which is useless in Unity but still present by default.) * 6/10 could easily find and launch a game that wasn't in the launcher. (P2 deserves special mention for finding and launching the game's .desktop file amongst piles of detritus in Nautilus's File System search results.) * Only 1/9 (P4) easily added that game to the launcher. For the other eight, the main problems were that the launcher disappeared when a window was maximized or at the left of the screen, that Dash items didn't have context menus, and that the launcher often didn't accept a drop. * 2/2 successfully removed an item from the launcher. * Only 2/6 noticed an XChat Gnome notification, despite (1) a notification bubble appearing, (2) the Ubuntu button going blue, (3) the messaging menu envelope going blue, and (4) an emblem appearing on XChat Gnome's launcher. * 9/9 easily launched LibreOffice Writer to write a letter. * 5/5 easily found today's date. * 9/9 easily saved their LibreOffice Writer document. (P1 recovered amazingly well after trying to save Letter to Mr Smith 08/04/11, and getting the vile response Error stating file '/home/ubuntu /Documents/Letter to Mr Smith 08/04': No such file or directory). * 9/11 people could easily close a window. The other two (P2, P7) were not the only ones to be attacked by a bug that hid the title bar for a window underneath the menu bar; they were just the only participants for whom that bug really cramped their style. * 9/9 easily found and opened an existing document. * 8/9 easily copied text from one document into another. The other one (P2) managed it eventually, but the missing title bar for one of the document windows was again a major stumbling block. * Only 1/7 (P9, a Windows 7 user) easily arranged windows side by side by discovering the window snap feature. (That's probably not really a problem;
Re: Default Desktop Experience for 11.04 - User testing results
On Fri, Apr 15, 2011 at 03:00:31AM +0100, Matthew Paul Thomas wrote: * 8/10 people could find a window's menus, but 7/8 of them learned to * Only 4/11 worked out how to change the background picture. * 6/10 could easily find and launch a game that wasn't in the * Only 1/9 (P4) easily added that game to the launcher. * 9/11 people could easily close a window. * 8/9 easily copied text from one document into another. * Only 5/10 could easily delete a document These seven items in particular seem like really basic tasks that ought to be testing at 90%, so these stats seem a lot lower than I'd expect. I'm curious whether these stats are higher/lower/same-as with Classic Desktop. IOW this needs a control group so we can tell if the new design brings improvement or regression. Also, these tests measure usability, but not their overall impression. Did they like it? Find it frustrating/confusing? Bryce -- ubuntu-desktop mailing list ubuntu-desktop@lists.ubuntu.com https://lists.ubuntu.com/mailman/listinfo/ubuntu-desktop
