Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Tilman Bayer]

On Fri, Sep 14, 2012 at 9:35 AM, Bence Damokos  wrote:
> Hi,
>
> On Fri, Sep 14, 2012 at 6:08 PM, Jan-bart de Vreede > wrote:
>
>> Hey
>>
>> So someone sent me the internal-l mail and I do think that the "zero
>> real-world" thing is taken out of context here. But a few points
>>

>> 3) I am sure that the data set allows us to see chapter's individual
>> responses, depending on whether or not we know the country (I would figure
>> we do?)
The survey asks indeed for the country where the respondent lives, and
using that data, we plan to evaluate the results of the performance
rating question a little differently this time. On Meta, I have said
more about the objections against the part asking about editors'
perception of chapters.

>>
> I believe the biggest issue from a methodological point of view, once you
> filter out responses from non-chapter countries, is that the question asks
> for a composite average rating on chapterS performance. There is doubt that
> the results obtained in this way would be valid or reliable measures of
> chapters ratings.
> (On the face of it, if one has to rate together a well-performing known
> organization and 38 unknown ones, it is unlikely that he would adjust his
> rating of the known organization upwards to get to an average, it seems
> more likely that he would do a downwards adjustment and perhaps vice-versa
> in case of a low-performing local organization; different people would
> apply different adjustments and it becomes doubtful whether we can ever get
> an accurate average from a simple question asking for opinion about 39
> organizations.
Interesting thoughts, but I did not see an argument here why a
downwards adjustment should be considered more likely than an upwards
adjustment. Anyway, every opinion survey, say the many Gallup polls,
has to deal with the basic fact that the respondents can have very
different levels of knowledge about the subject which they are asked
to rate. Rather than blindly ignoring this, many readers of opinion
surveys are actively aware of it and for example, are interested in
how opinions change when knowledge improves or decreases among
respondents. And yes, these opinions may be subjective, or unfair,
having been unduly influenced by isolated success stories or scandals.
(In our own opinion.)

Or to put it differently, with these ratings of individual chapters -
or indeed the rating of the Foundation in the same question - you
would likewise have the issue that respondents judge the whole
entity's work despite often knowing only small parts of it; any kind
of such question could be attacked for merely asking about a
"composite average" where parts are missing from the respondent's
knowledge. In reality, at one point one just has to define a certain
number as representing the respondent's opinion about a certain thing,
and measure it without second-guessing the (non-existing) formula by
which the respondent calculates it in their heads as an average of
other ratings.

> Asking individuals to rate one chapter if relevant and than taking the
> [weighed] average of the ratings from the 39 relevant countries would be
> more accurate perhaps. And as a control group, people from outside chapter
> countries could be asked to rate all chapters in one question to see if
> there is anything to learn there from the answers.)
>  In general, when a concept is complex, surveys tend to ask more questions
> to get a better picture, just as the WESI-score on Wikipedia satisfaction
> is a composite from a number of questions on this survey.
>
This is often called a latent variable
(https://en.wikipedia.org/wiki/Latent_variable ). It is used here
because some questions are difficult to ask directly for (mainly)
language reasons - e.g., "how satisfied are you with... " would be too
ambiguous.

On Fri, Sep 14, 2012 at 12:57 AM, charles andrès
 wrote:
> Dear Jan-Bart,
> Unfortunately the exact wordings was "I'll point out also that there are zero 
> real-world implications for the survey results".
>
> Because we all agree that there is now such thing like a zero real-world 
> impact survey, we really hope that the raw results of this survey will be 
> made as public as possible (privacy issue), and that in the future , survey 
> including question about WMF partners (chapters are not the only ones) will 
> be done since the very beginning in collaboration with all the partners 
> involved.

Yes, we agree about the benefits of making the anonymized raw data
from such surveys available to everyone. As it has been noted in the
Q&A on Meta since July, this will be done for this survey as it was
done for its two predecessors (see
http://dumps.wikimedia.org/other/surveys/ - would love to see more
such datasets from other surveys in the movement, btw, and more people
using them to do their own analyses).

Regarding the "real world": Of course in a strict interpretation,
everything written somewhere has real-world impact, even if it is just
to change the color o

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Frédéric Schütz]

On 14/09/12 18:08, Jan-bart de Vreede wrote:


4) Doesn't every survey contain questions that don't apply to the whole 
responding audience?


No, because the answer would not be terribly useful. In all surveys I 
have seen in my work life [statistics work -- I don't design much 
surveys myself, but reply to a lot of them from different horizons :-], 
this does never happen.


Typically, the filtering for a question about chapters would be 
something like this (simplified and written quickly):


1) Do you know any organization active within the Wikimedia movement ?
(free text fields allow people to enter names)

2) This page contain a list of organizations active within the Wikimedia 
movement; please tick all the ones you know (even if you have only heard 
the name)


(a list follows, with maybe "WMF", "Local chapters in general", and a 
list of individual chapters)


3) People are then asked to rate and/or comment each entity that was 
mentioned under (1) or (2).


This way, not only do we avoid having people give their opinion on a 
topic they've never heard about, but we also get two different levels of 
knowledge on the topic (either the user knew it well enough to list the 
name, or he had to be reminded).


Frédéric

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Bence Damokos]

Hi,

On Fri, Sep 14, 2012 at 6:08 PM, Jan-bart de Vreede  wrote:

> Hey
>
> So someone sent me the internal-l mail and I do think that the "zero
> real-world" thing is taken out of context here. But a few points
>
> 1) Lets not have these discussions on internal-l, there is no reason to
> not have those in public
> 2) See 1)
> 3) I am sure that the data set allows us to see chapter's individual
> responses, depending on whether or not we know the country (I would figure
> we do?)
>
I believe the biggest issue from a methodological point of view, once you
filter out responses from non-chapter countries, is that the question asks
for a composite average rating on chapterS performance. There is doubt that
the results obtained in this way would be valid or reliable measures of
chapters ratings.
(On the face of it, if one has to rate together a well-performing known
organization and 38 unknown ones, it is unlikely that he would adjust his
rating of the known organization upwards to get to an average, it seems
more likely that he would do a downwards adjustment and perhaps vice-versa
in case of a low-performing local organization; different people would
apply different adjustments and it becomes doubtful whether we can ever get
an accurate average from a simple question asking for opinion about 39
organizations.
Asking individuals to rate one chapter if relevant and than taking the
[weighed] average of the ratings from the 39 relevant countries would be
more accurate perhaps. And as a control group, people from outside chapter
countries could be asked to rate all chapters in one question to see if
there is anything to learn there from the answers.)
 In general, when a concept is complex, surveys tend to ask more questions
to get a better picture, just as the WESI-score on Wikipedia satisfaction
is a composite from a number of questions on this survey.

> 4) Doesn't every survey contain questions that don't apply to the whole
> responding audience?
>
Probably not - depends on the survey design and goals.

Best regards,
Bence

>
> Jan-Bart
>
>
>
>
> On 14 sep. 2012, at 17:05, Florence Devouard  wrote:
>
> >
> >
> > On 9/12/12 4:14 PM, Jan-Bart de Vreede wrote:
> >> Hey
> >>
> >> So I might have missed some mails on this thread (perhaps because they
> were not posted on this public list) but I highly doubt that Sue perform
> surveys that do not have a real-world impact on our operations. I know that
> the results of the previous surveys were used in several discussions
> (including at a board level) in order to provide more insight….
> >
> >
> > Indeed. I agree. These surveys do have real-world impact, which is why
> we objected to a survey asking people from all over the world how they
> would rate Wikimedia Chapters activities when
> > 1) there is likely no chapter in their country
> > 2) they may have no idea that a "wikimedia chapter" is for example
> "Wikimedia Washington DC" or "Wikimedia Israel"
> > 3) all chapters are collectively considered regardless of individual
> differences
> >
> > And since you comment on that specific sentence
> >
> > "I'll point out also that there are zero real-world implications for the
> survey results."
> >
> > I'd like to clarify that these exact words come from Sue herself in an
> email sent on the 10th of September on internal-l.
> >
> > I am glad to read that you disagree with that statement and recognize
> that there is real-world impact.
> >
> >
> > (did not want to comment any further on that problematic survey, but
> wanted to attribute statement properly)
> >
> >
> > Flo
> >
> >
> >> On the other hand, using these surveys to gain more insight is not the
> same as using them to "hold each other accountable" which is sometimes easy
> to do. Every survey (and questions) has a lots of interpretation magic
> which can easily lead you astray, but I don't have to tell you (the
> community) this :)
> >
> >
> >> Jan-Bart
> >>
> >>
> >> On 10 Sep 2012, at 23:01, Delphine Ménard <
> notafishz-re5jqeeqqe8avxtiumw...@public.gmane.org> wrote:
> >>
> >>> Contrarily to Sue, I do
> >>> think that these surveys (should) have a real-world impact and
> >>> (should) keep us all on our toes, fine tuned to the critisicism, needs
> >>> and wishes of the editors of the WIkimedia projects. As such I expect
> >>> us to make sure that we do get as precise a picture as possible of
> >>> what those are.
> >>
> >> ___
> >> Wikimedia-l mailing list
> >> wikimedia-l-rusutvdil2icgmh+5r0dm0b+6bgkl...@public.gmane.org
> >> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> >>
> >
> >
> >
> > ___
> > Wikimedia-l mailing list
> > Wikimedia-l@lists.wikimedia.org
> > Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
>
>
> ___
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/li

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Jan-bart de Vreede]

Hey

So someone sent me the internal-l mail and I do think that the "zero 
real-world" thing is taken out of context here. But a few points

1) Lets not have these discussions on internal-l, there is no reason to not 
have those in public
2) See 1)
3) I am sure that the data set allows us to see chapter's individual responses, 
depending on whether or not we know the country (I would figure we do?)
4) Doesn't every survey contain questions that don't apply to the whole 
responding audience?

Jan-Bart




On 14 sep. 2012, at 17:05, Florence Devouard  wrote:

> 
> 
> On 9/12/12 4:14 PM, Jan-Bart de Vreede wrote:
>> Hey
>> 
>> So I might have missed some mails on this thread (perhaps because they were 
>> not posted on this public list) but I highly doubt that Sue perform surveys 
>> that do not have a real-world impact on our operations. I know that the 
>> results of the previous surveys were used in several discussions (including 
>> at a board level) in order to provide more insight….
> 
> 
> Indeed. I agree. These surveys do have real-world impact, which is why we 
> objected to a survey asking people from all over the world how they would 
> rate Wikimedia Chapters activities when
> 1) there is likely no chapter in their country
> 2) they may have no idea that a "wikimedia chapter" is for example "Wikimedia 
> Washington DC" or "Wikimedia Israel"
> 3) all chapters are collectively considered regardless of individual 
> differences
> 
> And since you comment on that specific sentence
> 
> "I'll point out also that there are zero real-world implications for the 
> survey results."
> 
> I'd like to clarify that these exact words come from Sue herself in an email 
> sent on the 10th of September on internal-l.
> 
> I am glad to read that you disagree with that statement and recognize that 
> there is real-world impact.
> 
> 
> (did not want to comment any further on that problematic survey, but wanted 
> to attribute statement properly)
> 
> 
> Flo
> 
> 
>> On the other hand, using these surveys to gain more insight is not the same 
>> as using them to "hold each other accountable" which is sometimes easy to 
>> do. Every survey (and questions) has a lots of interpretation magic which 
>> can easily lead you astray, but I don't have to tell you (the community) 
>> this :)
> 
> 
>> Jan-Bart
>> 
>> 
>> On 10 Sep 2012, at 23:01, Delphine Ménard 
>>  wrote:
>> 
>>> Contrarily to Sue, I do
>>> think that these surveys (should) have a real-world impact and
>>> (should) keep us all on our toes, fine tuned to the critisicism, needs
>>> and wishes of the editors of the WIkimedia projects. As such I expect
>>> us to make sure that we do get as precise a picture as possible of
>>> what those are.
>> 
>> ___
>> Wikimedia-l mailing list
>> wikimedia-l-rusutvdil2icgmh+5r0dm0b+6bgkl...@public.gmane.org
>> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
>> 
> 
> 
> 
> ___
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l


___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Florence Devouard]

On 9/12/12 4:14 PM, Jan-Bart de Vreede wrote:

Hey

So I might have missed some mails on this thread (perhaps because they were not 
posted on this public list) but I highly doubt that Sue perform surveys that do 
not have a real-world impact on our operations. I know that the results of the 
previous surveys were used in several discussions (including at a board level) 
in order to provide more insight….



Indeed. I agree. These surveys do have real-world impact, which is why 
we objected to a survey asking people from all over the world how they 
would rate Wikimedia Chapters activities when

1) there is likely no chapter in their country
2) they may have no idea that a "wikimedia chapter" is for example 
"Wikimedia Washington DC" or "Wikimedia Israel"
3) all chapters are collectively considered regardless of individual 
differences


And since you comment on that specific sentence

"I'll point out also that there are zero real-world implications for the 
survey results."


I'd like to clarify that these exact words come from Sue herself in an 
email sent on the 10th of September on internal-l.


I am glad to read that you disagree with that statement and recognize 
that there is real-world impact.



(did not want to comment any further on that problematic survey, but 
wanted to attribute statement properly)



Flo



On the other hand, using these surveys to gain more insight is not the same as using them 
to "hold each other accountable" which is sometimes easy to do. Every survey 
(and questions) has a lots of interpretation magic which can easily lead you astray, but 
I don't have to tell you (the community) this :)




Jan-Bart


On 10 Sep 2012, at 23:01, Delphine Ménard 
 wrote:


Contrarily to Sue, I do
think that these surveys (should) have a real-world impact and
(should) keep us all on our toes, fine tuned to the critisicism, needs
and wishes of the editors of the WIkimedia projects. As such I expect
us to make sure that we do get as precise a picture as possible of
what those are.


___
Wikimedia-l mailing list
wikimedia-l-rusutvdil2icgmh+5r0dm0b+6bgkl...@public.gmane.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l





___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Samuel Klein]

On Fri, Sep 14, 2012 at 3:57 AM, charles andrès  wrote:


> Because we all agree that there is now such thing like a zero real-world
> impact survey, we really hope that the raw results of this survey will be
> made as public as possible (privacy issue), and that in the future , survey
> including question about WMF partners (chapters are not the only ones) will
> be done since the very beginning in collaboration with all the partners
> involved.
>

Yes.  What would be better ways to improve communication and collaboration?
 There were a couple of months of very gradual feedback before a recent
surge of interest close to the planned launch date.  The suggestions for
making these surveys regular, perhaps quarterly, would allow for continuous
participation.

And some parallel surveys prepared by different statisticians/analysts
might be useful as well.  There are economies of scale in consolidating
into a single survey, but there are also systemic biases that are hard to
avoid if there is *only* a single survey.  A few large surveys, judiciously
broadcast [not everyone has to see every survey announcement] might address
both issues.

SJ
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[charles andrès]

Dear Jan-Bart,
Unfortunately the exact wordings was "I'll point out also that there are zero 
real-world implications for the survey results".

Because we all agree that there is now such thing like a zero real-world impact 
survey, we really hope that the raw results of this survey will be made as 
public as possible (privacy issue), and that in the future , survey including 
question about WMF partners (chapters are not the only ones) will be done since 
the very beginning in collaboration with all the partners involved.

sincerely

Charles


___
Charles ANDRES, Chairman
"Wikimedia CH" – Association for the advancement of free knowledge –
www.wikimedia.ch
Skype: charles.andres.wmch
IRC://irc.freenode.net/wikimedia-ch



Le 12 sept. 2012 à 16:14, Jan-Bart de Vreede  a écrit :

> Hey
> 
> So I might have missed some mails on this thread (perhaps because they were 
> not posted on this public list) but I highly doubt that Sue perform surveys 
> that do not have a real-world impact on our operations. I know that the 
> results of the previous surveys were used in several discussions (including 
> at a board level) in order to provide more insight….
> 
> On the other hand, using these surveys to gain more insight is not the same 
> as using them to "hold each other accountable" which is sometimes easy to do. 
> Every survey (and questions) has a lots of interpretation magic which can 
> easily lead you astray, but I don't have to tell you (the community) this :)
> 
> Jan-Bart
> 
> 
> On 10 Sep 2012, at 23:01, Delphine Ménard  wrote:
> 
>> Contrarily to Sue, I do
>> think that these surveys (should) have a real-world impact and
>> (should) keep us all on our toes, fine tuned to the critisicism, needs
>> and wishes of the editors of the WIkimedia projects. As such I expect
>> us to make sure that we do get as precise a picture as possible of
>> what those are.
> 
> ___
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.org
> Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l
> 

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Jan-Bart de Vreede]

Hey

So I might have missed some mails on this thread (perhaps because they were not 
posted on this public list) but I highly doubt that Sue perform surveys that do 
not have a real-world impact on our operations. I know that the results of the 
previous surveys were used in several discussions (including at a board level) 
in order to provide more insight….

On the other hand, using these surveys to gain more insight is not the same as 
using them to "hold each other accountable" which is sometimes easy to do. 
Every survey (and questions) has a lots of interpretation magic which can 
easily lead you astray, but I don't have to tell you (the community) this :)

Jan-Bart


On 10 Sep 2012, at 23:01, Delphine Ménard  wrote:

> Contrarily to Sue, I do
> think that these surveys (should) have a real-world impact and
> (should) keep us all on our toes, fine tuned to the critisicism, needs
> and wishes of the editors of the WIkimedia projects. As such I expect
> us to make sure that we do get as precise a picture as possible of
> what those are.

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Delphine Ménard]

Hello Tilman,


On Sun, Sep 9, 2012 at 7:24 PM, Tilman Bayer  wrote:
> Hi Delphine,
>
> On Tue, Aug 21, 2012 at 6:57 AM, Delphine Ménard  wrote:
>> On Tue, Aug 21, 2012 at 3:32 PM, Tilman Bayer  wrote:
> ...
>>>
>>> Still, I was aware that there had been some objections to that
>>> question by chapter representatives (which I don't assume have to do
>>> with the fact that respondents rated chapters' performance lower than
>>> that of other entities in the two previous surveys), and looked into
>>> these concerns while the present questionnaire was prepared; I also
>>> reached out to one of the critics in person at Wikimania. But I still
>>> haven't seen a compelling argument why the way the question is asked
>>> should be biased against chapters. The argument that the opinion of
>>> Wikimedians who live in countries without chapter should not count
>>> seems weak to me, e.g. because the projects that the work of chapters
>>> aims to support are international, and because the question asked
>>> about chapters in general, not one particular chapter.
>>
>> That is not the argument I was trying to make (ie. voices of
>> Wikimedians in a country without chapter don't count). Rather, there
>> is a long list of things the Foundation does, where people are asked
>> whether they knew about it, or not. And after that, right when people
>> have been made aware of everything the Foundation does, they are asked
>> to rate the work of the Foundation. The same question about the
>> chapters comes after absolutely nothing has been said about chapter
>> work, which, I believe, does introduce a bias. In short, people are
>> being asked to rate something they *at this point in the survey* have
>> an idea about (for the WMF) although they might have had no idea about
>> it before starting the survey.
>> All I'm asking is that we review the context in which this question is
>> being asked so results make more sense.
> OK, after some other people also remarked that preceding this question
> by other questions which conveyed quite some information about the
> Foundation's activities but not about the chapters' activities. we
> have now rearranged the questions so that this is no longer the case.
>
> This is a bit of a compromise regarding the structuring of the
> questionnaire into sections, but fortunately it could be done without
> invalidating existing translations or changing the variables of the
> resulting dataset.

Thanks. I will not hide that I am still not sure whether we don't now
have two "out of context questions" instead of just one, but I guess
it's what we could do for this round, so thank you for doing this.

I sincerely hope that we can all together revisit this part of the
survey to give results that can be used by all of us to increase
satisfaction and performance in the future. Contrarily to Sue, I do
think that these surveys (should) have a real-world impact and
(should) keep us all on our toes, fine tuned to the critisicism, needs
and wishes of the editors of the WIkimedia projects. As such I expect
us to make sure that we do get as precise a picture as possible of
what those are.

Best,

Delphine
-- 
@notafish

NB. This gmail address is used for mailing lists. Personal emails will get lost.
Intercultural musings: Ceci n'est pas une endive - http://blog.notanendive.org
Photos with simple eyes: notaphoto - http://photo.notafish.org

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Craig Franklin]

Great, THANKYOU for finally moving on this.

Cheers,
Craig Franklin

Date: Sun, 9 Sep 2012 10:24:39 -0700
> From: Tilman Bayer 
> To: Wikimedia Mailing List 
> Subject: Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and
>     Office Hour
> Message-ID:
>  kijpcvrr6pdeb0usz_+_6q2yiraxr3fqz9ql...@mail.gmail.com>
> Content-Type: text/plain; charset=ISO-8859-1
>
> Hi Delphine,
>
> On Tue, Aug 21, 2012 at 6:57 AM, Delphine M?nard 
> wrote:
> > On Tue, Aug 21, 2012 at 3:32 PM, Tilman Bayer 
> wrote:
> ...
> >>
> >> Still, I was aware that there had been some objections to that
> >> question by chapter representatives (which I don't assume have to do
> >> with the fact that respondents rated chapters' performance lower than
> >> that of other entities in the two previous surveys), and looked into
> >> these concerns while the present questionnaire was prepared; I also
> >> reached out to one of the critics in person at Wikimania. But I still
> >> haven't seen a compelling argument why the way the question is asked
> >> should be biased against chapters. The argument that the opinion of
> >> Wikimedians who live in countries without chapter should not count
> >> seems weak to me, e.g. because the projects that the work of chapters
> >> aims to support are international, and because the question asked
> >> about chapters in general, not one particular chapter.
> >
> > That is not the argument I was trying to make (ie. voices of
> > Wikimedians in a country without chapter don't count). Rather, there
> > is a long list of things the Foundation does, where people are asked
> > whether they knew about it, or not. And after that, right when people
> > have been made aware of everything the Foundation does, they are asked
> > to rate the work of the Foundation. The same question about the
> > chapters comes after absolutely nothing has been said about chapter
> > work, which, I believe, does introduce a bias. In short, people are
> > being asked to rate something they *at this point in the survey* have
> > an idea about (for the WMF) although they might have had no idea about
> > it before starting the survey.
> > All I'm asking is that we review the context in which this question is
> > being asked so results make more sense.
> OK, after some other people also remarked that preceding this question
> by other questions which conveyed quite some information about the
> Foundation's activities but not about the chapters' activities. we
> have now rearranged the questions so that this is no longer the case.
>
> This is a bit of a compromise regarding the structuring of the
> questionnaire into sections, but fortunately it could be done without
> invalidating existing translations or changing the variables of the
> resulting dataset.
>
> Also, the launch of the survey had been postponed into this month for
> various reasons, including allowing more time to respond to feedback
> like this.
> --
> Tilman Bayer
> Senior Operations Analyst (Movement Communications)
> Wikimedia Foundation
> IRC (Freenode): HaeB
>
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Tilman Bayer]

Hi Delphine,

On Tue, Aug 21, 2012 at 6:57 AM, Delphine Ménard  wrote:
> On Tue, Aug 21, 2012 at 3:32 PM, Tilman Bayer  wrote:
...
>>
>> Still, I was aware that there had been some objections to that
>> question by chapter representatives (which I don't assume have to do
>> with the fact that respondents rated chapters' performance lower than
>> that of other entities in the two previous surveys), and looked into
>> these concerns while the present questionnaire was prepared; I also
>> reached out to one of the critics in person at Wikimania. But I still
>> haven't seen a compelling argument why the way the question is asked
>> should be biased against chapters. The argument that the opinion of
>> Wikimedians who live in countries without chapter should not count
>> seems weak to me, e.g. because the projects that the work of chapters
>> aims to support are international, and because the question asked
>> about chapters in general, not one particular chapter.
>
> That is not the argument I was trying to make (ie. voices of
> Wikimedians in a country without chapter don't count). Rather, there
> is a long list of things the Foundation does, where people are asked
> whether they knew about it, or not. And after that, right when people
> have been made aware of everything the Foundation does, they are asked
> to rate the work of the Foundation. The same question about the
> chapters comes after absolutely nothing has been said about chapter
> work, which, I believe, does introduce a bias. In short, people are
> being asked to rate something they *at this point in the survey* have
> an idea about (for the WMF) although they might have had no idea about
> it before starting the survey.
> All I'm asking is that we review the context in which this question is
> being asked so results make more sense.
OK, after some other people also remarked that preceding this question
by other questions which conveyed quite some information about the
Foundation's activities but not about the chapters' activities. we
have now rearranged the questions so that this is no longer the case.

This is a bit of a compromise regarding the structuring of the
questionnaire into sections, but fortunately it could be done without
invalidating existing translations or changing the variables of the
resulting dataset.

Also, the launch of the survey had been postponed into this month for
various reasons, including allowing more time to respond to feedback
like this.
-- 
Tilman Bayer
Senior Operations Analyst (Movement Communications)
Wikimedia Foundation
IRC (Freenode): HaeB

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[charles andrès]

Le 22 août 2012 à 13:11, Cristian Consonni  a écrit :

> 2012/8/21 Delphine Ménard :
>> If, however, we're going to mix editor's experience and satisfaction
>> about Wikimedia, I am cruelly missing any kind of feedback question
>> about the work of the chapters and/or other organisations or groups in
>> the Wikimedia Universe that would give people the right scope about
>> what is happening in a more "offline" kind of way. Of course, we could
>> do a separate survey for chapters, but if we're truly an international
>> movement, then all Wikimedia entities that support/interact with the
>> community probably would benefit from being put in the same bag in
>> order to fine tune their support and help for the Wikimedia
>> communities.
> 
> +1.
> 
+1 and I emphasize the "Wikimedia Communities" because according to the survey 
it seems that the Foundation is still believing that there is only one and 
unique community! 

charles



___
Charles ANDRES, Chairman
"Wikimedia CH" – Association for the advancement of free knowledge –
www.wikimedia.ch
Skype: charles.andres.wmch
IRC://irc.freenode.net/wikimedia-ch



___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Cristian Consonni]

2012/8/21 Delphine Ménard :
> If, however, we're going to mix editor's experience and satisfaction
> about Wikimedia, I am cruelly missing any kind of feedback question
> about the work of the chapters and/or other organisations or groups in
> the Wikimedia Universe that would give people the right scope about
> what is happening in a more "offline" kind of way. Of course, we could
> do a separate survey for chapters, but if we're truly an international
> movement, then all Wikimedia entities that support/interact with the
> community probably would benefit from being put in the same bag in
> order to fine tune their support and help for the Wikimedia
> communities.

+1.

Cristian

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Delphine Ménard]

On Tue, Aug 21, 2012 at 3:32 PM, Tilman Bayer  wrote:
> I'm not sure she actually said that. In any case, it is wrong -
> question D6 in the last survey asked if there was a Wikimedia chapter
> in the country where the respondent lived
> (https://meta.wikimedia.org/w/index.php?title=File:December_2011_Wikipedia_Editor_Survey_topline.pdf&page=5
> ).

My bad, glad it was there. And yes she did. Maybe not in those exact
words, but she did put the results into context.
>
>> This was already quite criticized last time, yet
>> the question is still there, unchanged, and without more context than
>> it had last time. (under FINAL THOUGHTS).
> We tried to avoid modifying questions in order to preserve consistence
> and be able  to do some longitudinal analysis (as it was already begun
> for that question in http://blog.wikimedia.org/?p=14296 ).

So if a question is poorly phrased, we'll continue having it till the
end of time to preserve consistency? Mind you, I do want that question
in, I just want it within the same context frame that is given to the
same question about the Foundation. And I'm also missing a question
about other entities that might actually help Wikimedians that we're
or we're not aware of.

>
> Still, I was aware that there had been some objections to that
> question by chapter representatives (which I don't assume have to do
> with the fact that respondents rated chapters' performance lower than
> that of other entities in the two previous surveys), and looked into
> these concerns while the present questionnaire was prepared; I also
> reached out to one of the critics in person at Wikimania. But I still
> haven't seen a compelling argument why the way the question is asked
> should be biased against chapters. The argument that the opinion of
> Wikimedians who live in countries without chapter should not count
> seems weak to me, e.g. because the projects that the work of chapters
> aims to support are international, and because the question asked
> about chapters in general, not one particular chapter.

That is not the argument I was trying to make (ie. voices of
Wikimedians in a country without chapter don't count). Rather, there
is a long list of things the Foundation does, where people are asked
whether they knew about it, or not. And after that, right when people
have been made aware of everything the Foundation does, they are asked
to rate the work of the Foundation. The same question about the
chapters comes after absolutely nothing has been said about chapter
work, which, I believe, does introduce a bias. In short, people are
being asked to rate something they *at this point in the survey* have
an idea about (for the WMF) although they might have had no idea about
it before starting the survey.
All I'm asking is that we review the context in which this question is
being asked so results make more sense.

>
> In any case, in parallel to preparing the upcoming survey, we have
> recently been compiling the public dataset with the anonymized
> responses from the last survey, which was uploaded yesterday here:
> http://dumps.wikimedia.org/other/surveys/editorsurveynov-dec2011/
> Using this, anyone can do the analysis you suggest and check if
> ratings differed significantly in chapter/non-chapter countries.

That's great. Thanks. For the record, I'm not expecting the results to
be so extremely different, but I think the fact that they might be or
might not be is extremely important to know.



>> Finally, what I regret most, is that o little time was allocated to
>> reviewing this survey collaboratively. There was about 20 days to
>> review, comment, give feedback, translate etc. and this in the mist of
>> the summer holiday for a big part of our community. When I see such
>> processes as the ToS revision [1] conducted successfully allowing for
>> 120 days of discussion, and such surveys rushed through the summer,
>> when their data will probably be used to decide much of the strategic
>> orientation of the next year, I'm thinking we're missing out on trying
>> to collect data that is truly relevant to the work we do and that
>> helps us to review our orientations and adjust how well we support our
>> editing community.

> Of course there are lots of interesting and important questions that
> had to be left out of this survey. As said earlier, the idea is to run
> the editor survey more frequently from now on, probably quarterly and
> in a more lightweight version, with a different focus in each.


That's good news and I hope the collaborative process can be
reinforced and more time is allowed for comments, reviews, changes and
finetuning.

Best,

Delphine

-- 
@notafish

NB. This gmail address is used for mailing lists. Personal emails will get lost.
Intercultural musings: Ceci n'est pas une endive - http://blog.notanendive.org
Photos with simple eyes: notaphoto - http://photo.notafish.org

___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Tilman Bayer]

On Tue, Aug 21, 2012 at 5:04 AM, Delphine Ménard  wrote:
> Hi, sorry to fish out a very old message, but since the survey is
> about to go live, I would like to share some concerns I have about it.
> Unfortunately, I was on holidays as this announcement came out, so
> couldn't do it earlier.
>
> I find the idea of an editor's survey to be extremely important, since
> it is (among other) a good indicator of how the editing community
> perceives the atmosphere in the projects, the evolution of the
> software and such things. However, I feel this survey is a bit of a
> missed opportunity on different aspects.
>
> There is a mix of "feedback about the projects and the community" and
> "satisfaction about the WMF", which does not, in my opinion, quite fit
> together. I find we should separate those things so as to keep people
> free of personal opinions about what the organisation may or may not
> do for/with them and let them focus better on their editor's
> experience as such. Moreover, this would allow for more questions
> about editing, maybe a short presentation of new tools, rating them
> etc. which seems to be quite absent from this questionnaire. For
> example, I would love to see a question in the technology part about
> whether people want/edit from their mobile device, or if they are
> familiar with the mobile apps and use them, that kind of stuff.
> (rationale given for taking these questions out was length of the
> survey, but I think these things are much more relevant to the
> well-editing of the contributors than how well they rate the WMF
> work). Not to mention that trying to get some feedback from sister
> projects would be good also (Commons is already a good first step).
>
> If, however, we're going to mix editor's experience and satisfaction
> about Wikimedia, I am cruelly missing any kind of feedback question
> about the work of the chapters and/or other organisations or groups in
> the Wikimedia Universe that would give people the right scope about
> what is happening in a more "offline" kind of way. Of course, we could
> do a separate survey for chapters, but if we're truly an international
> movement, then all Wikimedia entities that support/interact with the
> community probably would benefit from being put in the same bag in
> order to fine tune their support and help for the Wikimedia
> communities.
>
> Sue in Washington at Wikimania pulled out the results of one question
> that was asked in the editor's survey about whether people were
> satisfied about the work of the Foundation and the work of the
> chapters. She underlined herself that the results to this question
> were probably difficult to interpret out of the box since at no point
> in the survey was there a question about whether people were aware
> that there was a chapter in their country, which would have qulified
> the results a bit.
I'm not sure she actually said that. In any case, it is wrong -
question D6 in the last survey asked if there was a Wikimedia chapter
in the country where the respondent lived
(https://meta.wikimedia.org/w/index.php?title=File:December_2011_Wikipedia_Editor_Survey_topline.pdf&page=5
).

> This was already quite criticized last time, yet
> the question is still there, unchanged, and without more context than
> it had last time. (under FINAL THOUGHTS).
We tried to avoid modifying questions in order to preserve consistence
and be able  to do some longitudinal analysis (as it was already begun
for that question in http://blog.wikimedia.org/?p=14296 ).

Still, I was aware that there had been some objections to that
question by chapter representatives (which I don't assume have to do
with the fact that respondents rated chapters' performance lower than
that of other entities in the two previous surveys), and looked into
these concerns while the present questionnaire was prepared; I also
reached out to one of the critics in person at Wikimania. But I still
haven't seen a compelling argument why the way the question is asked
should be biased against chapters. The argument that the opinion of
Wikimedians who live in countries without chapter should not count
seems weak to me, e.g. because the projects that the work of chapters
aims to support are international, and because the question asked
about chapters in general, not one particular chapter.

In any case, in parallel to preparing the upcoming survey, we have
recently been compiling the public dataset with the anonymized
responses from the last survey, which was uploaded yesterday here:
http://dumps.wikimedia.org/other/surveys/editorsurveynov-dec2011/
Using this, anyone can do the analysis you suggest and check if
ratings differed significantly in chapter/non-chapter countries.


> In short people are asked to
> rate the Foundation about everything it does, while the chapters are
> never mentionned, and then people are asked to rate the work of both.
> Interesting way to look at it.
>
> We're doing much better, but there is still some English Wik

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Delphine Ménard]

Hi, sorry to fish out a very old message, but since the survey is
about to go live, I would like to share some concerns I have about it.
Unfortunately, I was on holidays as this announcement came out, so
couldn't do it earlier.

I find the idea of an editor's survey to be extremely important, since
it is (among other) a good indicator of how the editing community
perceives the atmosphere in the projects, the evolution of the
software and such things. However, I feel this survey is a bit of a
missed opportunity on different aspects.

There is a mix of "feedback about the projects and the community" and
"satisfaction about the WMF", which does not, in my opinion, quite fit
together. I find we should separate those things so as to keep people
free of personal opinions about what the organisation may or may not
do for/with them and let them focus better on their editor's
experience as such. Moreover, this would allow for more questions
about editing, maybe a short presentation of new tools, rating them
etc. which seems to be quite absent from this questionnaire. For
example, I would love to see a question in the technology part about
whether people want/edit from their mobile device, or if they are
familiar with the mobile apps and use them, that kind of stuff.
(rationale given for taking these questions out was length of the
survey, but I think these things are much more relevant to the
well-editing of the contributors than how well they rate the WMF
work). Not to mention that trying to get some feedback from sister
projects would be good also (Commons is already a good first step).

If, however, we're going to mix editor's experience and satisfaction
about Wikimedia, I am cruelly missing any kind of feedback question
about the work of the chapters and/or other organisations or groups in
the Wikimedia Universe that would give people the right scope about
what is happening in a more "offline" kind of way. Of course, we could
do a separate survey for chapters, but if we're truly an international
movement, then all Wikimedia entities that support/interact with the
community probably would benefit from being put in the same bag in
order to fine tune their support and help for the Wikimedia
communities.

Sue in Washington at Wikimania pulled out the results of one question
that was asked in the editor's survey about whether people were
satisfied about the work of the Foundation and the work of the
chapters. She underlined herself that the results to this question
were probably difficult to interpret out of the box since at no point
in the survey was there a question about whether people were aware
that there was a chapter in their country, which would have qulified
the results a bit. This was already quite criticized last time, yet
the question is still there, unchanged, and without more context than
it had last time. (under FINAL THOUGHTS). In short people are asked to
rate the Foundation about everything it does, while the chapters are
never mentionned, and then people are asked to rate the work of both.
Interesting way to look at it.

We're doing much better, but there is still some English Wikipedia
centrism in Question F2 for example. ;-)
: How well do you believe the Foundation supports:
English Wikipedia?
Wikipedia sites in other languages?


Finally, what I regret most, is that so little time was allocated to
reviewing this survey collaboratively. There was about 20 days to
review, comment, give feedback, translate etc. and this in the mist of
the summer holiday for a big part of our community. When I see such
processes as the ToS revision [1] conducted successfully allowing for
120 days of discussion, and such surveys rushed through the summer,
when their data will probably be used to decide much of the strategic
orientation of the next year, I'm thinking we're missing out on trying
to collect data that is truly relevant to the work we do and that
helps us to review our orientations and adjust how well we support our
editing community.

Best,

Delphine



[1] http://meta.wikimedia.org/wiki/Talk:Terms_of_use

On Sat, Jul 28, 2012 at 7:28 AM, Christine Moellenberndt
 wrote:
> *Hi everyone,
>
> It's been a bit since I last emailed this list (or any list, for that
> matter!)... you may remember me, I worked at the Foundation last year in the
> Community Department, working with Philippe on any number of issues, as well
> as with the OTRS team.  I've come back to work on a short term project with
> the Foundation, and I have to say it's great to be back! (and a great break
> from my Master's thesis!)
>
> We're getting ready to run the next version of the Editor Survey, for August
> 2012. This will be the third incarnation we've run since 2011.  As with the
> prior incarnations of the survey, we'll be looking at a variety of topics,
> this time with the goal of not only understanding your needs and pressing
> issues while interacting with fellow editors, but also focusing on editors'
> satisfaction with the wor

Re: [Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Tilman Bayer]

On Fri, Jul 27, 2012 at 10:28 PM, Christine Moellenberndt <
christine...@gmail.com> wrote:

> *Hi everyone,
>
> It's been a bit since I last emailed this list (or any list, for that
> matter!)... you may remember me, I worked at the Foundation last year in
> the Community Department, working with Philippe on any number of issues, as
> well as with the OTRS team.  I've come back to work on a short term project
> with the Foundation, and I have to say it's great to be back! (and a great
> break from my Master's thesis!)
>
> We're getting ready to run the next version of the Editor Survey, for
> August 2012. This will be the third incarnation we've run since 2011.  As
> with the prior incarnations of the survey, we'll be looking at a variety of
> topics, this time with the goal of not only understanding your needs and
> pressing issues while interacting with fellow editors, but also focusing on
> editors' satisfaction with the work of the Foundation.
>
> The last time we ran an editor survey, it was completed by over 6,000
> respondents.  When you break that down, it means that each minute of time
> demanded by the survey corresponds to 100 hours of Wikipedians' time.  We
> want to make sure that this time is spent wisely, ensuring that the
> questions we have are worded clearly, don't cause confusion, and will
> generate meaningful answers.  So we'd like to ask you to take a look at the
> survey, and give us feedback on the questions.  You can find them here:
>
> https://meta.wikimedia.org/**wiki/Research:Wikipedia_**
> Editor_Survey_August_2012/**Questions
>  Wikipedia_Editor_Survey_**August_2012/Questions
> >
> ... and please leave your feedback on the talk page there so we can keep
> the discussion in one place :)
>
> You can find out more information about the survey here:
>
> https://meta.wikimedia.org/**wiki/Research:Wikipedia_**
> Editor_Survey_August_2012
>
> Also, we are planning an IRC Office Hour on the survey, this **Tuesday,
> July 31 at 1700 UTC.** (See https://meta.wikimedia.org/**
> wiki/IRC_office_hoursforgeneral
>  information about IRC Office hours)
>
Just a quick reminder that this is happening two hours from now.

>
> I know there has been some discussion about offering Office Hours in a
> broader range of times, and I know this time may not be the greatest for
> some... but this was the best time we could find currently.
>
> Thanks everyone!!
>
> -Christine
> Wikimedia Foundation*
> __**_
> Wikimedia-l mailing list
> Wikimedia-l@lists.wikimedia.**org 
> Unsubscribe: 
> https://lists.wikimedia.org/**mailman/listinfo/wikimedia-l
>



-- 
Tilman Bayer
Senior Operations Analyst (Movement Communications)
Wikimedia Foundation
IRC (Freenode): HaeB
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l

[Wikimedia-l] Upcoming Survey, Feedback requested, and Office Hour

[Christine Moellenberndt]

*Hi everyone,

It's been a bit since I last emailed this list (or any list, for that 
matter!)... you may remember me, I worked at the Foundation last year in 
the Community Department, working with Philippe on any number of issues, 
as well as with the OTRS team.  I've come back to work on a short term 
project with the Foundation, and I have to say it's great to be back! 
(and a great break from my Master's thesis!)


We're getting ready to run the next version of the Editor Survey, for 
August 2012. This will be the third incarnation we've run since 2011. 
 As with the prior incarnations of the survey, we'll be looking at a 
variety of topics, this time with the goal of not only understanding 
your needs and pressing issues while interacting with fellow editors, 
but also focusing on editors' satisfaction with the work of the Foundation.


The last time we ran an editor survey, it was completed by over 6,000 
respondents.  When you break that down, it means that each minute of 
time demanded by the survey corresponds to 100 hours of Wikipedians' 
time.  We want to make sure that this time is spent wisely, ensuring 
that the questions we have are worded clearly, don't cause confusion, 
and will generate meaningful answers.  So we'd like to ask you to take a 
look at the survey, and give us feedback on the questions.  You can find 
them here:


https://meta.wikimedia.org/wiki/Research:Wikipedia_Editor_Survey_August_2012/Questions

... and please leave your feedback on the talk page there so we can keep 
the discussion in one place :)


You can find out more information about the survey here:

https://meta.wikimedia.org/wiki/Research:Wikipedia_Editor_Survey_August_2012

Also, we are planning an IRC Office Hour on the survey, this **Tuesday, 
July 31 at 1700 UTC.** (See 
https://meta.wikimedia.org/wiki/IRC_office_hoursfor general information 
about IRC Office hours)


I know there has been some discussion about offering Office Hours in a 
broader range of times, and I know this time may not be the greatest for 
some... but this was the best time we could find currently.


Thanks everyone!!

-Christine
Wikimedia Foundation*
___
Wikimedia-l mailing list
Wikimedia-l@lists.wikimedia.org
Unsubscribe: https://lists.wikimedia.org/mailman/listinfo/wikimedia-l