On Thu, Dec 12, 2013 at 12:26 PM, Gregory Zobel <[email protected]> wrote: > Hi Rob, > > The purpose of some of the text was to provide some framing explanation. I > have removed that text from this iteration and included the suggested > changes. > > I think it might be wise to run this first with the mailing list members > that you mentioned. That way we could get results from regular users. Based > on their responses, we could then hone down and improve the survey for use > with new visitors. > > Best, > gz > > ============================= > Here's the revised version: > > Proposed Core SUS Usability Questions to Evaluate User Experience with the > Apache Open Office Website [Draft] > > INTRO framing text: [I don’t know enough about AOO to write this bit up] >
I can draft this part. > > > Questions > Part 1 > I think that I would like to use this website frequently. > I found the website unnecessarily complex. > I thought the website was easy to use. > I think that I would need the support of a technical person to be able to > use this website. > I found the various functions in this website were well integrated. > I thought there was too much inconsistency in this website. > I would imagine that most people would learn to use this website very > quickly. > I found the website very cumbersome to use. > I felt very confident using the website. > I needed to learn a lot of things before I could get going with this > website. > > Response options: > 1 Strongly Disagree to 5 Strongly Agree > > > Part 2 > What was the purpose of your visit today? > Learn about Apache OpenOffice > Download Apache OpenOffice > Solve a problem with OpenOffice > Learn more about the OpenOffice open source community > Other (please describe) > x user fills in their own response > This one may or may not end up in the survey. If we put a link to the survey in the website header, then it would be a good question, since the person is visiting the website for a particular purpose. But if we send the survey link out to the mailing list then the question doesn't really have that kind of use context. > > What do you like most about the Apache Open Office website? > The forums > The online manuals > ...add a couple more > x user fills in their own response > > What frustrates you most about the Apache Open Office website? > Navigation > The help manuals > ...add a couple more > x user fills in their own response > We already have detailed information on what pages users visit most, how much time is spent on pages, what the landing pages are, what the exist pages are, etc. This is from Google Analytics. So maybe we should focus the "like most" and "frustrates" items on higher level qualities (ease of navigation, speed, etc.) rather than particular content areas like help manuals? Or maybe we want both? > If there was one thing you could add to the Apache Open Office website, > what would it be? > ...add a couple more > x user fills in their own response > > Thank you for providing your responses and your feedback. Your responses > will help us make AOO a better and stronger community. If you are willing, > we would appreciate a bit more information about you, your computing > preferences, and how you use AOO. > > Demographic Questions > > Gender: F M prefer to not answer > Age range: 18-22 23-30 31-35 36-40 41-45 46-50 51-55 56-60 61-70 70+ > prefer to not answer May need "under 18" or "13-18" as well. > Operating system: Windows Mac Linux Other prefer to not answer > Browser: Firefox Chrome Safari IE Other prefer to not answer > Version of Open Office: prefer to not answer > Industry in which you work: Government Education Industry Entrepreneur > Creative prefer to not answer > Where you are located (continent? country?) dropdown menu of options? > prefer to not answer > Language in which you use AOO? dropdown menu of options? prefer to not > answer > > > Again, thank you for sharing your time, thoughts and responses. As you > know, AOO is as committed to open source as we are to privacy. We are also > interested in being as transparent with our growth, processes, and data as > possible. With your permission, we would like to publish and share your > responses with our community. We won’t share it without your permission. To > remain anonymous and not share your data, you don’t need to do anything > else. But if you are willing to share your responses, please tick the box > below. > > Sharing Data Checkbox > I agree to share my data/responses. [And whatever legalese is needed.] > This looks good. I will draft a survey in LimeSurvey and send a link to the list so we can review and fine tune. Regards, -Rob > > > On Thu, Dec 12, 2013 at 7:27 AM, Rob Weir <[email protected]> wrote: > >> On Wed, Dec 11, 2013 at 11:57 PM, Gregory Zobel <[email protected]> wrote: >> > Hi Folks, >> > >> > Based on some comments and a short exchange on the dev list, I've >> developed >> > a rough draft for a UX survey. The goal is to solicit user feedback about >> > how usable the AAO site is. To help frame the results and grasp who the >> > users are, several qualitative and demographic questions are included. >> > >> > I'm brand new to the AOO community, so I apologize if I've overlooked >> > something obvious. >> > >> > Below is the proposed survey. I have brief explanatory comments between >> > each small set of questions. >> > >> > I do hope this is useful or provides a viable foundation for a future UX >> > review. >> > >> > Best, >> > gz >> > >> > >> > *********************** >> > *********************** >> > >> > Proposed Core SUS Usability Questions to Evaluate User Experience with >> the >> > Apache Open Office Website [Draft] >> > >> > The purpose of this text is to: >> > propose a body of questions for use in a usability test of the AOO site; >> > provide support and explanation for these questions; >> > identify important issues to help us frame the data. >> > >> > To measure user experience and user response to the AOO website, it is >> > important to use reliable tools to gather data. Fortunately, a variety of >> > such tools exist. One optimal tool is the System Usability Scale which >> John >> > Brooke released in 1986. >> > >> >> I assume the above is background info for us, and we'd have a >> different prologue text for the respondent to read in the actual >> survey? >> >> > The SUS is a 10 item questionnaire with 5 response options. >> > >> > SUS Questions >> > I think that I would like to use this system frequently. >> > I found the system unnecessarily complex. >> > I thought the system was easy to use. >> > I think that I would need the support of a technical person to be able to >> > use this system. >> > I found the various functions in this system were well integrated. >> > I thought there was too much inconsistency in this system. >> > I would imagine that most people would learn to use this system very >> > quickly. >> > I found the system very cumbersome to use. >> > I felt very confident using the system. >> > I needed to learn a lot of things before I could get going with this >> system. >> > >> > Response options: >> > 1 Strongly Disagree to 5 Strongly Agree >> > >> >> Is it important to use this language exactly? Or can we replace "this >> system" with "this website"? The risk of "this system" is some >> fraction of the users may misinterpret the questions as being a survey >> about the OpenOffice product or even about their Browser! If we can >> be explicit it would help, I think. >> >> >> > source: http://www.measuringusability.com/sus.php >> > [A number of other sources can be references, and SUS is an industry >> > standard.] >> > >> > SUS Strengths: >> > The system is tested and is reliable. >> > The data produced is pretty clean, and there are numerous parties able to >> > analyze and evaluate the data. >> > The brevity is useful for users because it does not take too much of >> their >> > time. >> > It takes far less time to identify, code, and gather Likert-type response >> > than open-ended questions. >> > >> > SUS Weaknesses: >> > Results produced only enable system-wide results, i.e. it does not help >> > identify specific or local problems. >> > Results are strictly numerical. >> > Users are not able to give voice to their own opinions or share accurate >> or >> > useful insights--user can only click a box. >> > >> > >> > Proposed Addition to SUS: >> > In addition to the SUS scale, I propose having four or five additional >> > qualitative questions which allow users to give more rich, personalized, >> > specific, or custom feedback. This can be useful in terms of identifying >> > specific problems, demonstrating interest in users’ opinions, and >> > identifying gaps that those in the AOO culture are too close to to >> identify. >> > >> > These kinds of questions, however, can generate huge volumes of >> responses. >> > And often they are different. Evaluating and coding this information can >> > drive you nuts. One way to address this potential problem is to frame the >> > open-ended questions and then provide tick boxes for the three or five >> most >> > common responses. >> > >> >> Two things that LimeSurvey will collect for us automatically: >> >> 1) The IP address of the respondent. I think we just use this to >> check for abusive multiple submissions and then discard. >> >> 2) The time it took to respond to the survey, in seconds. Not sure >> how useful this information is, but there might be some lower >> threshold beneath which we might suspect the survey was not actually >> read. >> >> 2) The "referrer" page that lead them to the survey. This might be >> quite useful. We have the ability to add a link to the survey on >> every page of the website, by modifying the template. >> >> But a general concern: We know from Google Analytics that around 80% >> of visits are from users visiting the website for the first time. The >> remaining 20% are repeat visitors. For first-time visitors when do >> they take the survey? If they see a survey link and respond to it >> when they first see it (say the home page) then they are responding >> based on a very limited experience with the website, perhaps only >> visiting one page. Of course, others might respond to the survey >> after drilling into the website more. >> >> The Google survey service had the ability to delay the presentation of >> the survey option until the user had already visited, say, 4 pages. >> I'm not sure how they did that and how we can do the same. >> >> Another option would be to survey repeat visitors. We could do that >> by not putting the survey on the website but sending it to our mailing >> list of around 10,000 users. That would give a different angle. >> >> It all depends on what we want: new users? experienced users? A mix/ >> >> >> > Additional Questions: >> > What do you like most about the Apache Open Office website? >> > <options> >> > The forums >> > The online manuals >> > ...add a couple more >> > x user fills in their own response >> > >> >> The above is an example of something a first-time visitor might not be >> able to answer. >> >> >> > What frustrates you most about the Apache Open Office website? >> > <options> >> > Navigation >> > The help manuals >> > ...add a couple more >> > x user fills in their own response >> > >> > If there was one thing you could add to the Apache Open Office website, >> > what would it be? >> > <options> >> > ...add a couple more >> > x user fills in their own response >> > >> >> Perhaps also: "What was the purpose of your visit today?" >> >> Learn about Apache OpenOffice >> Download Apache OpenOffice >> Solve a problem with OpenOffice >> Learn more about the OpenOffice open source community >> Other (please describe) >> >> The idea is that the usability of the system might vary depending on >> the intended use, we might find it works well for some things, but >> less well for other things. >> >> > >> > Potential transitional text for users: >> > Thank you for providing your responses and your feedback. Your responses >> > will help us make AOO a better and stronger community. If you are >> willing, >> > we would appreciate a bit more information about you, your computing >> > preferences, and how you use AOO. >> > >> > >> > Additional Demographic Questions >> > We probably want to keep these limited, otherwise they get tedious and >> > drive users away. >> > Also, we should include a “prefer not to answer” as an option for each of >> > the questions. >> > >> > >> > Gender: F M prefer to not answer >> > Age range: 18-22 23-30 31-35 36-40 41-45 46-50 51-55 56-60 61-70 70+ >> > prefer to not answer >> > Operating system: Windows Mac Linux Other prefer to not answer >> > Browser: Firefox Chrome Safari IE Other prefer to not answer >> > Version of Open Office: prefer to not answer >> > Industry in which you work: Government Education Industry >> Entrepreneur >> > Creative prefer to not answer >> > Where you are located (continent? country?) dropdown menu of options? >> > prefer to not answer >> > Language in which you use AOO? dropdown menu of options? prefer to not >> > answer >> > >> > >> > More Transitional text for users: >> > Again, thank you for sharing your time, thoughts and responses. As you >> > know, AOO is as committed to open source as we are to privacy. We are >> also >> > interested in being as transparent with our growth, processes, and data >> as >> > possible. With your permission, we would like to publish and share your >> > responses with our community. We won’t share it without your permission. >> To >> > remain anonymous and not share your data, you don’t need to do anything >> > else. But if you are willing to share your responses, please tick the box >> > below. >> > >> > >> > Sharing Data Checkbox: >> > I agree to share my data/responses. [And whatever legalese is needed.] >> > >> > >> > -- >> > --- >> > Gregory B. Zobel, Ph.D. >> > Assistant Professor of Educational Technology >> > MSEd Program Coordinator >> > >> > Western Oregon University >> > 345 N. Monmouth Ave >> > Monmouth, OR 97361 >> >> --------------------------------------------------------------------- >> To unsubscribe, e-mail: [email protected] >> For additional commands, e-mail: [email protected] >> >> > > > -- > --- > Gregory B. Zobel, Ph.D. > Assistant Professor of Educational Technology > MSEd Program Coordinator > > Western Oregon University > 345 N. Monmouth Ave > Monmouth, OR 97361 --------------------------------------------------------------------- To unsubscribe, e-mail: [email protected] For additional commands, e-mail: [email protected]
