Hello Stan, sorry for getting back so late, but I was ensnared in yet another user study (with some different focus) in China for the last few weeks. Hopefully, I will be able to post some results we got as a "side-effect" of this new study soon. I will try to address the issues you mentioned, also considering our latest findings:
1) In the Omelette project, we have created several means to add widgets directly to a workspace using recommendations. While the quality of recommendations is a tricky topic, users like the ability to add new widgets without leaving their workspace. One tractable option for this might be a search field on the top of the page offering a direct result list so I can click a widget to add it to the workspace. I am inspired by the twitter search field here, which offers icons and additional data on the result (e.g., type of result) immediately. On the other hand, virtually all of our users did not expect the page layout to be restricted to some fixed number of columns (that has to be edited separately). Nearly all of them tried to instinctively drag&drop widgets like icons on a desktop or views in Eclipse and became frustrated when they could not do so. So, the default layout of a page should be something that dynamically allocates space (maybe using a dynamic number of columns). 2) I am very glad to see that you agree on the feedback usability issues. One particular idea that I have had is that upon opening a workspace, the widget frames should be overlayed by the big widget icon until they have loaded and some "loading" indicator. This way, users will immediately know that their browser has not crashed and where which widget is. 3) Thank you for your concerns regarding the formalties of the report. This was the first usability report I have ever created and I will be glad to improve the structure of future reports with your hints. I have now heard of Techsmith Morae for the first time and it looks very promising. In our studies, however, we have also used "discount" approaches (Google Docs questionaires, measuring times by stopwatch, recording screen, audio and camera using the tool XSplit). All the best, Matthias -----Ursprüngliche Nachricht----- Von: Drozdetski, Stan A. [mailto:[email protected]] Gesendet: Dienstag, 13. November 2012 23:15 An: [email protected] Cc: Matthias Niederhausen Betreff: RE: Usability evaluation report Matthias, thank you for sharing this with the community! Very helpful to see not just the summary of findings, but also your entire protocol. I'm picking out two major issue groups in need of improvement: 1) Navigation to/from widget store + workflow for adding gadgets In our internal testing, we've seen quite a wide variety of mental models that people approach gadget-based interfaces with. Even folks with experience in other portal systems may not instinctively know which parts of the page are modifiable, and how to adjust the page contents vs page layout. We've tried video tutorials and first-time-use hints, but can't quite claim a silver bullet just yet. Another challenge is the separation between the widget store and the actual layout; it simply puts a barrier in the way of direct interface manipulation. I've been thinking through some solutions that show the user what's available without taking them off the page, but screen real estate becomes an issue. If you or anyone else have seen innovative solutions, please do share with the list. 2) Feedback for actions Yes, no feedback (or feedback that the user can't see) is a usability death spell. Same goes for long loading time. Thank you for noting that, and I agree with your recommendations 100%. Couple of unsolicited thoughts on the report itself - in case you're interested in what worked/didn't work for us at MITRE, or if there are other usability nerds on the list: - We find it very helpful to sort findings by severity (saving the per-task findings for the protocol section). Attaching recommendations, as you have done, is very helpful. - At least in the US, everyone is very sensitive about protecting the identity of participants. We try to take out any personally-identifiable information - even gender - out of all parts of the report, and refer to participants as P1, P2, and so on. - For studies with small to medium numbers of participants, we prefer to use ratios (3 out 10 users) in reporting, instead of percentages (30%). If anyone still has questions about sample sizes compared to traditional marketing approaches - well, there's an educational opportunity. Your notes and time-on-task measurements are very detailed - what recording methods do you use? We used to be very heavy into TechSmith Morae, but with the time demands of agile projects find ourselves moving more and more towards discount usability methods... Stan Drozdetski MITRE
