Re: [Wikitech-l] Your questions and votes for our CTO and VP of Product Q

2017-01-06 Thread Quim Gil 
After 25 days, 40 questions, and 2480 votes, the results are...
https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program/Wikimedia_Foundation_Product_and_Technology_Q%26A

Thank you to all participants. This experiment has been very interesting so
far. The Q session will happen next Tuesday at 9:30am Pacific, and there
will be a live-broadcast.

https://www.mediawiki.org/wiki/Wikimedia_Developer_Summit/2017/Program#Tuesday.2C_January_10th


On Tue, Jan 3, 2017 at 10:05 AM, Quim Gil  wrote:

> Hi,
>
> http://www.allourideas.org/wikidev17-product-technology-questions is
> closed for new questions but still open for votes until the end of
> Thursday. A couple of new questions were added in the past days. Please
> contribute a couple of minutes submitting some more votes!
>
> The results so far: http://www.allourideas.org/wikidev17-product-
> technology-questions/results
>
> On Sat, Dec 31, 2016 at 3:26 AM, Gergo Tisza  wrote:
>
>> On Tue, Dec 20, 2016 at 12:45 AM, Quim Gil  wrote:
>>
>> > The questions for this session are being crowdsourced at
>> > http://www.allourideas.org/wikidev17-product-technology-questions.
>> Anyone
>> > can propose questions and vote, anonymously, as many times as you want.
>> At
>> > the moment, we have 25 questions and 451 votes.
>> >
>> > An important technical detail: questions posted later have also good
>> > chances to make it to the top of the list as long as new voters select
>> > them. The ranking is made out of comparisons between questions, not
>> > accumulation of votes. For instance, the current top question is in fact
>> > one of the last that has been submitted so far.
>> >
>>
>> Right now the top question has a score of 70 based on 88 votes; the second
>> question has a score of 67 based on 1 vote. (This is not some super-rare
>> accident, either: number 8 and 9 on the popularity list both have 4
>> votes.)
>>
>
> Right now the top 10 have questions that have received as low as 8-15
> votes and as high as 80-101. These numbers will be more balanced if/when
> more people vote this week.
>
> I will not attempt to make a big fuss over participation theories, but
> IMHO Wikimedia processes are quite biased towards What Is Said By Who Talks
> First. This is a humble and harmless experiment in a different direction.
> While seeing a question with eight votes among the top 10 defies the
> traditional democracy paradigm, it also means that an idea that came later
> had any chance over those who were submitted early on.
>
> At the end what counts is the final result of the experiment. Regardless
> of the numbers, I think the current list makes sense, and I in fact it has
> been making sense all along since its second day or so.
>
>
>> That means that the scores can be heavily underspecified (ie. mostly
>> result
>> from the random numbers generated by their algorithm and not actual votes)
>
>
> Well, I am not sure. If a question with eight votes is among the top ten,
> it probably means that  it has been systematically preferred over other
> questions scoring similarly high.
>
> Currently the very last question has only two votes, which means that the
> same algorithm that can put new questions in the top segment can also bury
> them down.
>
> The solution to these potential biases is simple: more opinions submitted
> by more people, which is the basis of any healthy group participation.
>
> Gergo, I am not saying you are wrong (you have clearly done more research
> than myself). I am just saying that I don't think choosing this tool for
> this purpose was a wrong idea either.  :)
>
> --
> Quim Gil
> Engineering Community Manager @ Wikimedia Foundation
> http://www.mediawiki.org/wiki/User:Qgil
>



-- 
Quim Gil
Engineering Community Manager @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Your questions and votes for our CTO and VP of Product Q

2017-01-03 Thread Quim Gil 
Hi,

http://www.allourideas.org/wikidev17-product-technology-questions is closed
for new questions but still open for votes until the end of Thursday. A
couple of new questions were added in the past days. Please contribute a
couple of minutes submitting some more votes!

The results so far:
http://www.allourideas.org/wikidev17-product-technology-questions/results

On Sat, Dec 31, 2016 at 3:26 AM, Gergo Tisza  wrote:

> On Tue, Dec 20, 2016 at 12:45 AM, Quim Gil  wrote:
>
> > The questions for this session are being crowdsourced at
> > http://www.allourideas.org/wikidev17-product-technology-questions.
> Anyone
> > can propose questions and vote, anonymously, as many times as you want.
> At
> > the moment, we have 25 questions and 451 votes.
> >
> > An important technical detail: questions posted later have also good
> > chances to make it to the top of the list as long as new voters select
> > them. The ranking is made out of comparisons between questions, not
> > accumulation of votes. For instance, the current top question is in fact
> > one of the last that has been submitted so far.
> >
>
> Right now the top question has a score of 70 based on 88 votes; the second
> question has a score of 67 based on 1 vote. (This is not some super-rare
> accident, either: number 8 and 9 on the popularity list both have 4 votes.)
>

Right now the top 10 have questions that have received as low as 8-15 votes
and as high as 80-101. These numbers will be more balanced if/when more
people vote this week.

I will not attempt to make a big fuss over participation theories, but IMHO
Wikimedia processes are quite biased towards What Is Said By Who Talks
First. This is a humble and harmless experiment in a different direction.
While seeing a question with eight votes among the top 10 defies the
traditional democracy paradigm, it also means that an idea that came later
had any chance over those who were submitted early on.

At the end what counts is the final result of the experiment. Regardless of
the numbers, I think the current list makes sense, and I in fact it has
been making sense all along since its second day or so.


> That means that the scores can be heavily underspecified (ie. mostly result
> from the random numbers generated by their algorithm and not actual votes)


Well, I am not sure. If a question with eight votes is among the top ten,
it probably means that  it has been systematically preferred over other
questions scoring similarly high.

Currently the very last question has only two votes, which means that the
same algorithm that can put new questions in the top segment can also bury
them down.

The solution to these potential biases is simple: more opinions submitted
by more people, which is the basis of any healthy group participation.

Gergo, I am not saying you are wrong (you have clearly done more research
than myself). I am just saying that I don't think choosing this tool for
this purpose was a wrong idea either.  :)

-- 
Quim Gil
Engineering Community Manager @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

Re: [Wikitech-l] Your questions and votes for our CTO and VP of Product Q

2016-12-30 Thread Gergo Tisza 
On Tue, Dec 20, 2016 at 12:45 AM, Quim Gil  wrote:

> The questions for this session are being crowdsourced at
> http://www.allourideas.org/wikidev17-product-technology-questions. Anyone
> can propose questions and vote, anonymously, as many times as you want. At
> the moment, we have 25 questions and 451 votes.
>
> An important technical detail: questions posted later have also good
> chances to make it to the top of the list as long as new voters select
> them. The ranking is made out of comparisons between questions, not
> accumulation of votes. For instance, the current top question is in fact
> one of the last that has been submitted so far.
>

Right now the top question has a score of 70 based on 88 votes; the second
question has a score of 67 based on 1 vote. (This is not some super-rare
accident, either: number 8 and 9 on the popularity list both have 4 votes.)
I argued that All Our Ideas is too experimental to be relied on back when
it was considered as the voting tool for an early iteration of what ended
up being the Community Tech Wishlist, and I still think that's the case.

The way their voting system works is that they assume each idea has some
appeal (an arbitrary real number) for each voter, the appeals for a given
idea are normally distributed, and when a voter is shown a question pair,
their probability of voting a given way is a certain function of the
difference in appeals. They then use various statistical methods to come up
with random values for the appeals which match the observed votes, and
using those values they can calculate the probability for each question
that a randomly selected voter would prefer that question to a randomly
selected alternative; those probabilities are used to score the questions.

That means that the scores can be heavily underspecified (ie. mostly result
from the random numbers generated by their algorithm and not actual votes)
for some questions; this is especially true for recently submitted
questions, which have a very small number of votes, so they will basically
get a random position in the ranking. As far as I can see, the journal
article [1] where they present their method doesn't discuss this problem at
all. This is not terribly useful as a real-world ranking model IMO, so I
hope that 1) there will be some human oversight when evaluating the
results, and 2) that we don't intend to use this system for any voting that
actually matters (getting weirdly prioritized results for a Q session is,
of course, not a huge deal).


[1] http://journals.plos.org/plosone/article?id=10.1371/journal.pone.0123483
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l

[Wikitech-l] Your questions and votes for our CTO and VP of Product Q

2016-12-20 Thread Quim Gil 
Hi,

On the second day of the Wikimedia Developer Summit (January 10) there will
be a Q session with Victoria Coleman (Wikimedia Foundation CTO) and Wes
Moran (VP of Product). It is a plenary session and it will be
video-streamed.

The questions for this session are being crowdsourced at
http://www.allourideas.org/wikidev17-product-technology-questions. Anyone
can propose questions and vote, anonymously, as many times as you want. At
the moment, we have 25 questions and 451 votes.

An important technical detail: questions posted later have also good
chances to make it to the top of the list as long as new voters select
them. The ranking is made out of comparisons between questions, not
accumulation of votes. For instance, the current top question is in fact
one of the last that has been submitted so far.

Why posting or voting a good question? One obvious reason is to encourage
the Foundation's Technology and Product top managers to bring a good answer
in a public session with minutes taken and video recording.  :) Beyond
that, if the ranking of questions makes sense and is backed by
participation numbers, it has a serious chance to influence plans and
discussions beyond the Summit.

The current ranking does make sense, but maybe you could help covering more
areas, other perspectives?

   1. How do we deal with the lack of maintainers for all Wikimedia
   deployed code?
   2. Do we have a plan to bring our developer documentation to the level
   of a top Internet website, a major free software project?
   3. For WMF dev teams, what is the right balance between pushing own work
   versus seeking and supporting volunteer contributors?
   4. During the next year or so, what balance do you think we should
   strike between new projects and technical debt?
   5. When are we going to work on a modern talk pages system for good?
   6. Whose responsibility is to assure that all MediaWiki core components
   and the extensions deployed in Wikimedia have active maintainers?
   7. How important is to have a well maintained and well promoted catalog
   of tools, apps, gadgets, bots, templates, extensions...?
   8. Will MediaWiki ever become easier to install and manage? (e.g. plugin
   manager à la Wordpress). How much do we care about enterprise users?
   9. What should be the role of the Architecture Committee in WMF planning
   (priorities, goals, resources...) and are we there yet?
   10. In addition to Community Tech, should the other WMF Product teams
   prioritize their work taking into account the Community Wishlist results?

The full list:
http://www.allourideas.org/wikidev17-product-technology-questions/results

-- 
Quim Gil
Engineering Community Manager @ Wikimedia Foundation
http://www.mediawiki.org/wiki/User:Qgil
___
Wikitech-l mailing list
Wikitech-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikitech-l