On Wed, Dec 11, 2013 at 11:57 PM, Gregory Zobel <[email protected]> wrote: > Hi Folks, > > Based on some comments and a short exchange on the dev list, I've developed > a rough draft for a UX survey. The goal is to solicit user feedback about > how usable the AAO site is. To help frame the results and grasp who the > users are, several qualitative and demographic questions are included. > > I'm brand new to the AOO community, so I apologize if I've overlooked > something obvious. > > Below is the proposed survey. I have brief explanatory comments between > each small set of questions. > > I do hope this is useful or provides a viable foundation for a future UX > review. > > Best, > gz > > > *********************** > *********************** > > Proposed Core SUS Usability Questions to Evaluate User Experience with the > Apache Open Office Website [Draft] > > The purpose of this text is to: > propose a body of questions for use in a usability test of the AOO site; > provide support and explanation for these questions; > identify important issues to help us frame the data. > > To measure user experience and user response to the AOO website, it is > important to use reliable tools to gather data. Fortunately, a variety of > such tools exist. One optimal tool is the System Usability Scale which John > Brooke released in 1986. >
I assume the above is background info for us, and we'd have a different prologue text for the respondent to read in the actual survey? > The SUS is a 10 item questionnaire with 5 response options. > > SUS Questions > I think that I would like to use this system frequently. > I found the system unnecessarily complex. > I thought the system was easy to use. > I think that I would need the support of a technical person to be able to > use this system. > I found the various functions in this system were well integrated. > I thought there was too much inconsistency in this system. > I would imagine that most people would learn to use this system very > quickly. > I found the system very cumbersome to use. > I felt very confident using the system. > I needed to learn a lot of things before I could get going with this system. > > Response options: > 1 Strongly Disagree to 5 Strongly Agree > Is it important to use this language exactly? Or can we replace "this system" with "this website"? The risk of "this system" is some fraction of the users may misinterpret the questions as being a survey about the OpenOffice product or even about their Browser! If we can be explicit it would help, I think. > source: http://www.measuringusability.com/sus.php > [A number of other sources can be references, and SUS is an industry > standard.] > > SUS Strengths: > The system is tested and is reliable. > The data produced is pretty clean, and there are numerous parties able to > analyze and evaluate the data. > The brevity is useful for users because it does not take too much of their > time. > It takes far less time to identify, code, and gather Likert-type response > than open-ended questions. > > SUS Weaknesses: > Results produced only enable system-wide results, i.e. it does not help > identify specific or local problems. > Results are strictly numerical. > Users are not able to give voice to their own opinions or share accurate or > useful insights--user can only click a box. > > > Proposed Addition to SUS: > In addition to the SUS scale, I propose having four or five additional > qualitative questions which allow users to give more rich, personalized, > specific, or custom feedback. This can be useful in terms of identifying > specific problems, demonstrating interest in users’ opinions, and > identifying gaps that those in the AOO culture are too close to to identify. > > These kinds of questions, however, can generate huge volumes of responses. > And often they are different. Evaluating and coding this information can > drive you nuts. One way to address this potential problem is to frame the > open-ended questions and then provide tick boxes for the three or five most > common responses. > Two things that LimeSurvey will collect for us automatically: 1) The IP address of the respondent. I think we just use this to check for abusive multiple submissions and then discard. 2) The time it took to respond to the survey, in seconds. Not sure how useful this information is, but there might be some lower threshold beneath which we might suspect the survey was not actually read. 2) The "referrer" page that lead them to the survey. This might be quite useful. We have the ability to add a link to the survey on every page of the website, by modifying the template. But a general concern: We know from Google Analytics that around 80% of visits are from users visiting the website for the first time. The remaining 20% are repeat visitors. For first-time visitors when do they take the survey? If they see a survey link and respond to it when they first see it (say the home page) then they are responding based on a very limited experience with the website, perhaps only visiting one page. Of course, others might respond to the survey after drilling into the website more. The Google survey service had the ability to delay the presentation of the survey option until the user had already visited, say, 4 pages. I'm not sure how they did that and how we can do the same. Another option would be to survey repeat visitors. We could do that by not putting the survey on the website but sending it to our mailing list of around 10,000 users. That would give a different angle. It all depends on what we want: new users? experienced users? A mix/ > Additional Questions: > What do you like most about the Apache Open Office website? > <options> > The forums > The online manuals > ...add a couple more > x user fills in their own response > The above is an example of something a first-time visitor might not be able to answer. > What frustrates you most about the Apache Open Office website? > <options> > Navigation > The help manuals > ...add a couple more > x user fills in their own response > > If there was one thing you could add to the Apache Open Office website, > what would it be? > <options> > ...add a couple more > x user fills in their own response > Perhaps also: "What was the purpose of your visit today?" Learn about Apache OpenOffice Download Apache OpenOffice Solve a problem with OpenOffice Learn more about the OpenOffice open source community Other (please describe) The idea is that the usability of the system might vary depending on the intended use, we might find it works well for some things, but less well for other things. > > Potential transitional text for users: > Thank you for providing your responses and your feedback. Your responses > will help us make AOO a better and stronger community. If you are willing, > we would appreciate a bit more information about you, your computing > preferences, and how you use AOO. > > > Additional Demographic Questions > We probably want to keep these limited, otherwise they get tedious and > drive users away. > Also, we should include a “prefer not to answer” as an option for each of > the questions. > > > Gender: F M prefer to not answer > Age range: 18-22 23-30 31-35 36-40 41-45 46-50 51-55 56-60 61-70 70+ > prefer to not answer > Operating system: Windows Mac Linux Other prefer to not answer > Browser: Firefox Chrome Safari IE Other prefer to not answer > Version of Open Office: prefer to not answer > Industry in which you work: Government Education Industry Entrepreneur > Creative prefer to not answer > Where you are located (continent? country?) dropdown menu of options? > prefer to not answer > Language in which you use AOO? dropdown menu of options? prefer to not > answer > > > More Transitional text for users: > Again, thank you for sharing your time, thoughts and responses. As you > know, AOO is as committed to open source as we are to privacy. We are also > interested in being as transparent with our growth, processes, and data as > possible. With your permission, we would like to publish and share your > responses with our community. We won’t share it without your permission. To > remain anonymous and not share your data, you don’t need to do anything > else. But if you are willing to share your responses, please tick the box > below. > > > Sharing Data Checkbox: > I agree to share my data/responses. [And whatever legalese is needed.] > > > -- > --- > Gregory B. Zobel, Ph.D. > Assistant Professor of Educational Technology > MSEd Program Coordinator > > Western Oregon University > 345 N. Monmouth Ave > Monmouth, OR 97361 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
