Re: GNOME user survey 2011 (v2)
On Fri, Aug 19, 2011 at 10:04 PM, Federico Mena Quintero feder...@ximian.com wrote: On Mon, 2011-08-01 at 18:35 +0300, Felipe Contreras wrote: What do you think? Keep in mind that Gnome 3 just hasn't been around for very long. Right now Gnome 3 is most likely only being used by technical people, Linux enthusiasts, etc. - it has not trickled down to end users yet. We may have found the top, obvious problems in Gnome 3, but not the long tail of bugs. Which is why I suggested to wait until 3.2. I think the survey would be much more useful with free-form answers, instead of fixed options. That way you may be able to eliminate one level of indirection (oh, most people said 'somewhat', now let's make another survey to find out why). While fixed options let you do amazing web hackery, free-form answers lead to more insightful results. From what I can see, you were dealing with a small numbers of responders, so free-form is excellent, but when you are talking about thousands of them, I don't think you can do very much. It might be a good idea to add a bit more of them, but I think the bulk should be fixed. * Being able to publish the raw results really helps; this way other people can help you extract statistics. Do ask permission from the respondents to post their replies so that other people can study them. I agree. * That survey probably gave too much importance to the system administrators themselves - not surprisingly, better admin tools was the most requested thing. However, the survey *did* get us good insight into end-user's problems. Makes sense. It may be very valuable to ask the heads of deployments what they think of Gnome 3 so far, even if they haven't evaluated it yet on their users - these people have a very good sense of what will work well for people in the real world. By the way, the main page for the deployments is this (I don't know how up-to-date it is): https://live.gnome.org/GnomeMarketing/GnomeDeployments About two years after that survey got published, some people had the idea of making it periodic - unfortunately I lost track of them and of that effort. You may want to look around for them; they'll have interesting thoughts, I'm sure. While that might be interesting, I am more interested on getting as many unfiltered results as possible. -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, 2011-08-01 at 18:35 +0300, Felipe Contreras wrote: What do you think? Keep in mind that Gnome 3 just hasn't been around for very long. Right now Gnome 3 is most likely only being used by technical people, Linux enthusiasts, etc. - it has not trickled down to end users yet. We may have found the top, obvious problems in Gnome 3, but not the long tail of bugs. I think the survey would be much more useful with free-form answers, instead of fixed options. That way you may be able to eliminate one level of indirection (oh, most people said 'somewhat', now let's make another survey to find out why). While fixed options let you do amazing web hackery, free-form answers lead to more insightful results. Back in 2006 I did a survey of the Gnome deployments (i.e. somewhere around the middle of the 2.x series). These questions were posted to various places: https://live.gnome.org/GnomeMarketing/QuestionsForDeployments And this is the final report with the results: http://people.gnome.org/~federico/docs/gnome-deployments-2006/index.html (At the end of the report there are links to the raw results that people sent in. Download them and read them; they are short and quite interesting. Where the links say, primates.ximian.com, please replace that with people.gnome.org - I can't re-generate the HTML report with that change just now, unfortunately.) Let me tell you about some things I learned from that survey: * Free-form answers work really well. It *will* take you some time to read them and see how to categorize or weigh them, but it's worth the effort. * Being able to publish the raw results really helps; this way other people can help you extract statistics. Do ask permission from the respondents to post their replies so that other people can study them. * That survey probably gave too much importance to the system administrators themselves - not surprisingly, better admin tools was the most requested thing. However, the survey *did* get us good insight into end-user's problems. * Those pie charts in the report make no sense. Apologies for chartjunk. It may be very valuable to ask the heads of deployments what they think of Gnome 3 so far, even if they haven't evaluated it yet on their users - these people have a very good sense of what will work well for people in the real world. By the way, the main page for the deployments is this (I don't know how up-to-date it is): https://live.gnome.org/GnomeMarketing/GnomeDeployments About two years after that survey got published, some people had the idea of making it periodic - unfortunately I lost track of them and of that effort. You may want to look around for them; they'll have interesting thoughts, I'm sure. Federico ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 6:46 PM, Alan Cox a...@lxorguk.ukuu.org.uk wrote: It doesn't tell you much about the environment Are you using it on Desktop Laptop Netbook Tablet Good point. Added. and perhaps a question about skill levels/years of computing experience. That might help understand which user categories want changes. Eg it's assumed that most of the 'customise it' requests are from experienced users but I don't think anyone has actualyl analysed that to find out ? That would be interesting. I'm not sure if other people would agree, but I'll add it on the next version and see. Cheers. -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
Hi, Here's the third version of the survey. === 01. Which of the following images best resemble your desktop? === (image selection) * GNOME 2 * GNOME 3 * Unity * KDE === 02. Overall, how happy are you with GNOME? === (single choice) * unhappy * not so happy * happy * very happy * completely ecstatic === 03. Where do you run GNOME? === (multiple choice, with other) + Desktop + Laptop + Netbook + Tablet === 04. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.x + I don't know + I'm not using it currently + other, please specify === 05. How long have you been using GNOME? (years) === (numeric) === 06. How do you compare your current GNOME version with the version from one year ago? === (single choice) * better * no changes * worse * cannot say === 07. Does GNOME do what you want? === (single choice) * Everything * Mostly * Somewhat * Barely * Not at all === 08. How happy are you with GNOME in regards to == (matrix) Columns: unhappy / not so happy / happy / very happy / completely ecstatic + ease of use + documentation + language availability + accessibility + community === 09. Which other desktop environments have you used in recent years? == (multiple choice, with other) + KDE + Unity + XFCE + LXDE + Enlightenment + other (please specify) === 10. How many years of experience do you have using computers? === (numeric) === 11. How often do you use terminal/console? == * What is that? * When I have no other option * I can't live without them * Is there anything else? === 12. Have you contributed to the GNOME project? === (single choice) * Yes * No === 13. Have you contacted the GNOME team? === (single choice) * Yes, successfully * No, I don't know how * No, never had the need === 14. If you could change three things in GNOME, what would they be? === (free form) === 15. Do you have any comments or suggestions for the GNOME team? === (free form) -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
Questions 2, 6, 7, and 8 are still leading. See my last email for a discussion of how to fix them. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Fri, Aug 5, 2011 at 8:06 PM, Jason D. Clinton m...@jasonclinton.com wrote: Questions 2, 6, 7, and 8 are still leading. See my last email for a discussion of how to fix them. Feel free to propose alternatives. -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Fri, Aug 5, 2011 at 12:09, Felipe Contreras felipe.contre...@gmail.com wrote: On Fri, Aug 5, 2011 at 8:06 PM, Jason D. Clinton m...@jasonclinton.com wrote: Questions 2, 6, 7, and 8 are still leading. See my last email for a discussion of how to fix them. Feel free to propose alternatives. I'm not writing this survey for you especially after penning an email explaining how to write non-leading survey questions and other considerations that you need to take in to account before proceeding. If you don't want to do this correctly then it would be better if you didn't do it all. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 7:08 PM, Bastien Nocera had...@hadess.net wrote: On Mon, 2011-08-01 at 18:35 +0300, Felipe Contreras wrote: After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? === 01. Which of the following images best resemble your desktop? === (images of various desktops) * GNOME 2 * GNOME 3 * Unity * KDE snip === 03. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.20 - 2.32 + 2.10 - 2.19 + 2.0 - 2.9 + pre 2.0 + not using it currently + other, please specify You expect the person answering the questionnaire to not be able to recognise GNOME in question 1 (and identify it from a picture) then be able to tell you which version they're using in question 3? I think people don't have any problem finding About in any system and get the version of the software. However, if your survey software has the option of adding notes on a question, we could add a bit of instructions. At least the online survey software that I found didn't have that option. -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 8:34 PM, Luc Pionchon pionchon@gmail.com wrote: On Mon, Aug 1, 2011 at 18:35, Felipe Contreras felipe.contre...@gmail.com wrote: After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? minor points below inline, Sorry if some points were already discussed in the previous long thread, just discard === 01. Which of the following images best resemble your desktop? === (images of various desktops) * GNOME 2 * GNOME 3 * Unity * KDE === 02. Overall, how happy are you with GNOME? === (single choice) * unhappy * not so happy * happy * very happy * completely ecstatic I would merge the two last ones, so positive and negative are balanced, and so there is no middle choice: * unhappy * not so happy * happy * completely happy That's fine by me, but I think it doesn't have a bigger spectrum. === 03. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.20 - 2.32 + 2.10 - 2.19 + 2.0 - 2.9 + pre 2.0 I would merge the above into: + 3.2 + 3.0 + 2.x Also fine by me. + not using it currently + other, please specify === 04. How long have you been using GNOME? (years) === (numeric) === 05. How do you compare your current GNOME version with the version from one year ago? === (single choice) * better * no changes * worse * cannot say assuming some persons answering may not use GNOME 3, wouldn't the results of this question be difficult to sort out? No? I don't see how. Either they will answer cannot say, or they would answer something that should be taken with a grain of salt. Fortunately those users would be identified. === 06. Does GNOME do what you want? === (single choice) * Everything * Mostly * Somewhat * Barely * Not at all === 07. How happy are you with GNOME in regards to == (matrix) Columns: unhappy / not so happy / happy / very happy / completely ecstatic about the columns, same remark as above. + ease of use + documentation + language availability + accessibility + distraction free I don't think that's category broad enough. === 08. Which other desktop environments have you used in recent years? == (multiple choice, with other) + KDE + Unity + XFCE + LXDE + Enlightenment + other (please specify) === 09. How often do you use terminal/console? == * What is that? I would clarify into: * I don't know what it is * I never use it * When I have no other option I think it is missing this category : * regularly for specific tasks Maybe. The purpose of this question wasn't to accurately identify the way they use the terminal, but just to get a feeling if they are geeks or not. * I can't leave without them * Is there anything else? === 10. Does GNOME include code or documentation by you? === or any contribution by you? I think there's more than just code and doc in GNOME. Actually I forgot to update this one: Does GNOME include contributions by you? (single choice) * Yes * No === 11. Have you contacted the GNOME team? === * Yes, successfuly s/fuly/fully/ ? True. * No, I don't know how * No, never had the need === 12. If you could change three things in GNOME what would they be? === (free form) === 13. Do you have any comments or suggestions for the GNOME team? === (free form) Cheers. -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
2011/8/2 Felipe Contreras felipe.contre...@gmail.com: On Mon, Aug 1, 2011 at 7:08 PM, Bastien Nocera had...@hadess.net wrote: On Mon, 2011-08-01 at 18:35 +0300, Felipe Contreras wrote: After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? === 01. Which of the following images best resemble your desktop? === (images of various desktops) * GNOME 2 * GNOME 3 * Unity * KDE snip === 03. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.20 - 2.32 + 2.10 - 2.19 + 2.0 - 2.9 + pre 2.0 + not using it currently + other, please specify You expect the person answering the questionnaire to not be able to recognise GNOME in question 1 (and identify it from a picture) then be able to tell you which version they're using in question 3? I think people don't have any problem finding About in any system and get the version of the software. However, if your survey software has the option of adding notes on a question, we could add a bit of instructions. At least the online survey software that I found didn't have that option. If the survey is started by a local / installed software, it should be smart enough to detect the running environment and not ask the user at all. -- Frederic Crozat ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 6:51 PM, jose.ali...@gmail.com jose.ali...@gmail.com wrote: On Mon, Aug 1, 2011 at 11:35 AM, Felipe Contreras felipe.contre...@gmail.com wrote: After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? === 06. Does GNOME do what you want? === (single choice) I would probably rephrase this to Can you accomplish all you want with GNOME? and add a text for the user to explain why? besides the single choice options. That doesn't achieve the same. Perhaps I can accomplish all I want, but perhaps only in a slow way, or something annoys me. I think Does GNOME do what you want? fits perfectly. === 09. How often do you use terminal/console? == * What is that? * When I have no other option * I can't leave without them I asume you mean I can't live without them Oops. -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
Hi, After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? === 01. Which of the following images best resemble your desktop? === (images of various desktops) * GNOME 2 * GNOME 3 * Unity * KDE === 02. Overall, how happy are you with GNOME? === (single choice) * unhappy * not so happy * happy * very happy * completely ecstatic === 03. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.20 - 2.32 + 2.10 - 2.19 + 2.0 - 2.9 + pre 2.0 + not using it currently + other, please specify === 04. How long have you been using GNOME? (years) === (numeric) === 05. How do you compare your current GNOME version with the version from one year ago? === (single choice) * better * no changes * worse * cannot say === 06. Does GNOME do what you want? === (single choice) * Everything * Mostly * Somewhat * Barely * Not at all === 07. How happy are you with GNOME in regards to == (matrix) Columns: unhappy / not so happy / happy / very happy / completely ecstatic + ease of use + documentation + language availability + accessibility === 08. Which other desktop environments have you used in recent years? == (multiple choice, with other) + KDE + Unity + XFCE + LXDE + Enlightenment + other (please specify) === 09. How often do you use terminal/console? == * What is that? * When I have no other option * I can't leave without them * Is there anything else? === 10. Does GNOME include code or documentation by you? === (single choice) * Yes * No === 11. Have you contacted the GNOME team? === * Yes, successfuly * No, I don't know how * No, never had the need === 12. If you could change three things in GNOME what would they be? === (free form) === 13. Do you have any comments or suggestions for the GNOME team? === (free form) -- Felipe Contreras ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
It doesn't tell you much about the environment Are you using it on Desktop Laptop Netbook Tablet and perhaps a question about skill levels/years of computing experience. That might help understand which user categories want changes. Eg it's assumed that most of the 'customise it' requests are from experienced users but I don't think anyone has actualyl analysed that to find out ? ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, 2011-08-01 at 18:35 +0300, Felipe Contreras wrote: Hi, After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? === 01. Which of the following images best resemble your desktop? === (images of various desktops) * GNOME 2 * GNOME 3 * Unity * KDE snip === 03. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.20 - 2.32 + 2.10 - 2.19 + 2.0 - 2.9 + pre 2.0 + not using it currently + other, please specify You expect the person answering the questionnaire to not be able to recognise GNOME in question 1 (and identify it from a picture) then be able to tell you which version they're using in question 3? ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 11:35 AM, Felipe Contreras felipe.contre...@gmail.com wrote: Hi, After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? === 06. Does GNOME do what you want? === (single choice) I would probably rephrase this to Can you accomplish all you want with GNOME? and add a text for the user to explain why? besides the single choice options. === 09. How often do you use terminal/console? == * What is that? * When I have no other option * I can't leave without them I asume you mean I can't live without them Greetings ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 18:35, Felipe Contreras felipe.contre...@gmail.com wrote: Hi, Hi, After going through all the feedback, here's the second version of the proposed survey. There is a proposal to delay the survey until 3.2 is released, to try to avoid some of the initial negative feedback of 3.0, I guess. If this is delayed for 3.2, perhaps there could be a small piece of software that pops up a notification for this, and perhaps some notifications in the future. What do you think? minor points below inline, Sorry if some points were already discussed in the previous long thread, just discard === 01. Which of the following images best resemble your desktop? === (images of various desktops) * GNOME 2 * GNOME 3 * Unity * KDE === 02. Overall, how happy are you with GNOME? === (single choice) * unhappy * not so happy * happy * very happy * completely ecstatic I would merge the two last ones, so positive and negative are balanced, and so there is no middle choice: * unhappy * not so happy * happy * completely happy === 03. Which GNOME version(s) are you using? === (multiple choice, with other) + 3.2 + 3.0 + 2.20 - 2.32 + 2.10 - 2.19 + 2.0 - 2.9 + pre 2.0 I would merge the above into: + 3.2 + 3.0 + 2.x + not using it currently + other, please specify === 04. How long have you been using GNOME? (years) === (numeric) === 05. How do you compare your current GNOME version with the version from one year ago? === (single choice) * better * no changes * worse * cannot say assuming some persons answering may not use GNOME 3, wouldn't the results of this question be difficult to sort out? No? === 06. Does GNOME do what you want? === (single choice) * Everything * Mostly * Somewhat * Barely * Not at all === 07. How happy are you with GNOME in regards to == (matrix) Columns: unhappy / not so happy / happy / very happy / completely ecstatic about the columns, same remark as above. + ease of use + documentation + language availability + accessibility + distraction free === 08. Which other desktop environments have you used in recent years? == (multiple choice, with other) + KDE + Unity + XFCE + LXDE + Enlightenment + other (please specify) === 09. How often do you use terminal/console? == * What is that? I would clarify into: * I don't know what it is * I never use it * When I have no other option I think it is missing this category : * regularly for specific tasks * I can't leave without them * Is there anything else? === 10. Does GNOME include code or documentation by you? === or any contribution by you? I think there's more than just code and doc in GNOME. (single choice) * Yes * No === 11. Have you contacted the GNOME team? === * Yes, successfuly s/fuly/fully/ ? * No, I don't know how * No, never had the need === 12. If you could change three things in GNOME what would they be? === (free form) === 13. Do you have any comments or suggestions for the GNOME team? === (free form) ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
Re: GNOME user survey 2011 (v2)
On Mon, Aug 1, 2011 at 10:35, Felipe Contreras felipe.contre...@gmail.com wrote: After going through all the feedback, here's the second version of the proposed survey. Surveys require careful consideration of the phrasing of questions and careful consideration of the selection bias of respondents. It doesn't look like you have considered either, even after the bias in phrasing of the questions was pointed out to you in the first round of emails. For example, here's a classic: Do you approve of the President's job on foreign affairs? versus: How would you rate the President's job on foreign affairs? The first question is leading by punching the survey taker in the face with a statement about their value judgement before you've even finished saying the sentence. The second asks the survey taker to make their own value judgement. Second, you have to carefully consider who is motivated to take a survey. People who are angry are much more likely to respond to a survey versus people who are happy. You can mitigate this problem by offering a survey that appears to have nothing to do with the subject matter that you're really looking for an answer on so that you get a truly random sampling of Linux users. You also must be careful not to recruit people to take the survey from communities which will contain angry people. For example, going to forums to find people to take a survey automatically selectively biases from people who were likely there to solve some kind of problem and are so already in a particular state of mind. There are hundreds of strategies to work around the selection bias problem. It *is* a good idea to run a survey but unless it's done carefully, you could actually do far more damage than good by reporting results which are not accurate or--worse--call in to question the motivations of the study. The survey you're proposing will require months of work to do correctly and is likely best achieved as a part of a large group effort or as part of an academic or non-profit funded venture. ___ desktop-devel-list mailing list desktop-devel-list@gnome.org http://mail.gnome.org/mailman/listinfo/desktop-devel-list
