Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Iolanda Pensa]

dear all
to collect strengths and weakness of this system of review you can add you 
comments in the discussion page 
https://wikimania2016.wikimedia.org/wiki/Talk:Critical_issues_presentations#Feedback_and_Evaluation
 
I have stated reporting your feedback, but please do not hesitate to correct, 
modify, add.

please consider
1. there are other kind of submissions: go for them! (we are updating them 
Monday February 8th) https://wikimania2016.wikimedia.org/wiki/Submissions. 
2. we are experimenting, surely to make Montreal better than us :) this is a 
clear objective of Wikimania Esino Lario 
https://meta.wikimedia.org/wiki/Wikimania_2016_bids/Esino_Lario/Evaluation
3. thank you to all reviewers and people who made submissions!!! 

iolanda/iopensa


> Il giorno 04 feb 2016, alle ore 09:22, WereSpielChequers 
>  ha scritto:
> 
> Two reviews per submission might work if we had a clear set of criteria that 
> the reviewers were following and sufficient training of the reviewers that 
> they were broadly consistent in their marking. But when you get the same 
> presentation being marked as 5 and 8, as one of mine was then the suspicion 
> is that the assessors are not working to the same criteria as each other. 
> That wouldn't matter so much if they were all assessing all submissions, 
> except that an assessor who varied between 0 and ten points would have far 
> more influence than assessors who usually voted 6, 7 or 8. But having that 
> level of inconsistency and only two reviews per submission makes the process 
> a lottery that depends on who the two reviewers are for your submission.
> 
> As for the content of the reviews, I don't consider that either "5 (average)" 
> or "6 (rather interesting) tell me anything as to why my submissions were 
> rejected.
> 
> The other two reviews at least managed one or two lines. One of them even 
> stretched to two sentences.
> 
> Hope Montreal manages something a bit better, I'm sure either Manilla or 
> Perth would have done.
> 
> 
> WereSpielChequers
> 
> 
> On 3 Feb 2016, at 23:22, Dariusz Jemielniak  > wrote:
> 
>> hi,
>> 
>> I have some comments as a person from Academia (and not involved in 
>> Wikimania process in any way):
>> 
>> 1. Short reviews are definitely not helping in addressing the frustration of 
>> rejection, yet are quite common in academic peer reviewing, especially for 
>> conferences. 
>> 
>> 2. Double blind peer review (not knowing who is reviewed, and not knowing 
>> who reviews) is a standard in Academia, although some perceive it as 
>> contributing to lack of responsibility (especially true in competitive 
>> journal submissions).
>> 
>> 3. Two reviewers per submission is absolutely on par with the conference 
>> standards I'm used to. Sometimes there are three, but two is absolutely 
>> acceptable (although a third opinion should be used if the two disagree too 
>> much). 
>> 
>> 4. It could be useful to sensitize the reviewers that the main purpose of 
>> the review is to help the author to do better next time. 
>> 
>> 5. All this is volunteer work. We should be, generally, grateful to 
>> reviewers (but in the same time grateful to the contributors, too). 
>> 
>> best,
>> 
>> dj
>> 
>> On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers > > wrote:
>> What kind of ridiculous process is this? This is all I got:
>> 
>> ===
>> 
>> --- REVIEW 1 -
>> PAPER: 194
>> TITLE: GLAM+Wikidata
>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>> 
>> OVERALL EVALUATION: 8 (Very good)
>> 
>> --- REVIEW ---
>> 8
>> 
>> 
>> --- REVIEW 2 -
>> PAPER: 194
>> TITLE: GLAM+Wikidata
>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>> 
>> OVERALL EVALUATION: 6 (Rather interesting)
>> 
>> --- REVIEW ---
>> 6
>> 
>> ==
>> 
>> So only two people reviewed this? Who are these people? Why is this secret? 
>> Last year I had 5 people reviewing my submission [1].
>> 
>> Maarten
>> 
>> [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5 
>> 
>> 
>> Op 3-2-2016 om 23:15 schreef Andy Mabbett:
>>> I've just received feedback on one of my pitches saying, in part:
>>> 
>>> "Bad boy Andy! This is supposed to be an anonymous review process, so 
>>> starting your abstract with your own name, is not entirely fair."
>>> 
>>> -- 
>>> Andy Mabbett
>>> @pigsonthewing
>>> http://pigsonthewing.org.uk 
>>> 
>>> ___
>>> Wikimania-l mailing list
>>> Wikimania-l@lists.wikimedia.org 
>>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l 
>>> 
>> 
>> 
>> 

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Sebastian Wallroth]

Hi Tomasz,

this is what actually happend. Please refer to
https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation


: In order to achieve the greatest possible neutrality, the submissions
will
: be evaluated online using a *double-blind peer-review process*. This
: means that two evaluators will review the submission without knowing
: the name of its author. *If there are strong divergences among the two
: evaluations, at least one other review will be made.*

Kind regards,
Sebastian

Am 04.02.2016 um 11:31 schrieb Tomasz Ganicz:
> Also - normally in Academia - if there are two strongly opposite
> reviews (one very positive, one very negative) a typical procedure is
> to send the submission to the third one.
>
> 2016-02-04 10:54 GMT+01:00 Tomasz Ganicz  >:
>
> Well I think that double blind peer review hardly make sense here,
> from the reviewer POV, as we are in fact small community and it
> is  easy to guess who was a submiter in most cases. For example -
> if there is a submission about project X in country Y, which was
> funded by WMF grant - it is very easy to find out who was grantee
> and it is rather obvious that that person is a submitter :-)
>
> Also  judging from the several reviewers comments which I saw
> already - they did not follow the very vague criteria which was
> posted here:
>
> 
> https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation
>
> Normally - at least in Academia - reviewers are forced directly
> (by the review form) to address their opinion in relation to the
> criteria. The criteria were:
>
> "
>
>  1. problems and possible solutions in a specific field
>  2. proposals for others to replicate
>  3. issues (positive or negative) which have emerged from projects
>  4. issues you want to raise which you feel have not been
> discussed yet
>  5. issues which are at the centre of an online debate that you
> would like to address offline
>
> "
>
> 1-4 are IMHO relatively easy to evaluate - I would expect from the
> reviews to answer yes or no to them. 5 is a bit tricky as it
> depends strongly of what the reviewer think is "at the centre"  -
> but I would expect that they at least explain in few words here
> what they think is "at the centre" or not :-)
>
>
>
> 2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak  >:
>
> hi,
>
> I have some comments as a person from Academia (and not
> involved in Wikimania process in any way):
>
> 1. Short reviews are definitely not helping in addressing the
> frustration of rejection, yet are quite common in academic
> peer reviewing, especially for conferences. 
>
> 2. Double blind peer review (not knowing who is reviewed, and
> not knowing who reviews) is a standard in Academia, although
> some perceive it as contributing to lack of responsibility
> (especially true in competitive journal submissions).
>
> 3. Two reviewers per submission is absolutely on par with the
> conference standards I'm used to. Sometimes there are three,
> but two is absolutely acceptable (although a third opinion
> should be used if the two disagree too much). 
>
> 4. It could be useful to sensitize the reviewers that the main
> purpose of the review is to help the author to do better next
> time. 
>
> 5. All this is volunteer work. We should be, generally,
> grateful to reviewers (but in the same time grateful to the
> contributors, too). 
>
> best,
>
> dj
>
> On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers
> > wrote:
>
> What kind of ridiculous process is this? This is all I got:
>
> ===
>
> --- REVIEW 1 -
> PAPER: 194
> TITLE: GLAM+Wikidata
> AUTHORS: Sandra Fauconnier and Maarten Dammers
>
> OVERALL EVALUATION: 8 (Very good)
>
> --- REVIEW ---
> 8
>
>
> --- REVIEW 2 -
> PAPER: 194
> TITLE: GLAM+Wikidata
> AUTHORS: Sandra Fauconnier and Maarten Dammers
>
> OVERALL EVALUATION: 6 (Rather interesting)
>
> --- REVIEW ---
> 6
>
> ==
>
> So only two people reviewed this? Who are these people? Why is 
> this secret? Last year I had 5 people reviewing my submission [1].
>
> Maarten
>
> [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5
>
>
> Op 3-2-2016 om 23:15 schreef Andy 

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Tomasz Ganicz]

Well I think that double blind peer review hardly make sense here, from the
reviewer POV, as we are in fact small community and it is  easy to guess
who was a submiter in most cases. For example - if there is a submission
about project X in country Y, which was funded by WMF grant - it is very
easy to find out who was grantee and it is rather obvious that that person
is a submitter :-)

Also  judging from the several reviewers comments which I saw already -
they did not follow the very vague criteria which was posted here:

https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation

Normally - at least in Academia - reviewers are forced directly (by the
review form) to address their opinion in relation to the criteria. The
criteria were:

"

   1. problems and possible solutions in a specific field
   2. proposals for others to replicate
   3. issues (positive or negative) which have emerged from projects
   4. issues you want to raise which you feel have not been discussed yet
   5. issues which are at the centre of an online debate that you would
   like to address offline

"

1-4 are IMHO relatively easy to evaluate - I would expect from the reviews
to answer yes or no to them. 5 is a bit tricky as it depends strongly of
what the reviewer think is "at the centre"  - but I would expect that they
at least explain in few words here what they think is "at the centre" or
not :-)


2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak :

> hi,
>
> I have some comments as a person from Academia (and not involved in
> Wikimania process in any way):
>
> 1. Short reviews are definitely not helping in addressing the frustration
> of rejection, yet are quite common in academic peer reviewing, especially
> for conferences.
>
> 2. Double blind peer review (not knowing who is reviewed, and not knowing
> who reviews) is a standard in Academia, although some perceive it as
> contributing to lack of responsibility (especially true in competitive
> journal submissions).
>
> 3. Two reviewers per submission is absolutely on par with the conference
> standards I'm used to. Sometimes there are three, but two is absolutely
> acceptable (although a third opinion should be used if the two disagree too
> much).
>
> 4. It could be useful to sensitize the reviewers that the main purpose of
> the review is to help the author to do better next time.
>
> 5. All this is volunteer work. We should be, generally, grateful to
> reviewers (but in the same time grateful to the contributors, too).
>
> best,
>
> dj
>
> On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers 
> wrote:
>
>> What kind of ridiculous process is this? This is all I got:
>>
>> ===
>>
>> --- REVIEW 1 -
>> PAPER: 194
>> TITLE: GLAM+Wikidata
>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>>
>> OVERALL EVALUATION: 8 (Very good)
>>
>> --- REVIEW ---
>> 8
>>
>>
>> --- REVIEW 2 -
>> PAPER: 194
>> TITLE: GLAM+Wikidata
>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>>
>> OVERALL EVALUATION: 6 (Rather interesting)
>>
>> --- REVIEW ---
>> 6
>>
>> ==
>>
>> So only two people reviewed this? Who are these people? Why is this secret? 
>> Last year I had 5 people reviewing my submission [1].
>>
>> Maarten
>>
>> [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5
>>
>>
>> Op 3-2-2016 om 23:15 schreef Andy Mabbett:
>>
>> I've just received feedback on one of my pitches saying, in part:
>>
>> "Bad boy Andy! This is supposed to be an anonymous review process, so
>> starting your abstract with your own name, is not entirely fair."
>>
>> --
>> Andy Mabbett
>> @pigsonthewing
>> http://pigsonthewing.org.uk
>>
>>
>> ___
>> Wikimania-l mailing 
>> listWikimania-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimania-l
>>
>>
>>
>> ___
>> Wikimania-l mailing list
>> Wikimania-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>>
>>
>
>
> --
>
> __
> prof. dr hab. Dariusz Jemielniak
> kierownik katedry Zarządzania Międzynarodowego
> i grupy badawczej NeRDS
> Akademia Leona Koźmińskiego
> http://n wrds.kozminski.edu.pl
>
> członek Akademii Młodych Uczonych Polskiej Akademii Nauk
> członek Komitetu Polityki Naukowej MNiSW
>
> Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An
> Ethnography of Wikipedia" (2014, Stanford University Press) mojego
> autorstwa http://www.sup.org/book.cgi?id=24010
>
> Recenzje
> Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml
> Pacific Standard:
> http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/
> Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia
> The Wikipedian:
> 

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Tomasz Ganicz]

Also - normally in Academia - if there are two strongly opposite reviews
(one very positive, one very negative) a typical procedure is to send the
submission to the third one.

2016-02-04 10:54 GMT+01:00 Tomasz Ganicz :

> Well I think that double blind peer review hardly make sense here, from
> the reviewer POV, as we are in fact small community and it is  easy to
> guess who was a submiter in most cases. For example - if there is a
> submission about project X in country Y, which was funded by WMF grant - it
> is very easy to find out who was grantee and it is rather obvious that that
> person is a submitter :-)
>
> Also  judging from the several reviewers comments which I saw already -
> they did not follow the very vague criteria which was posted here:
>
>
> https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation
>
> Normally - at least in Academia - reviewers are forced directly (by the
> review form) to address their opinion in relation to the criteria. The
> criteria were:
>
> "
>
>1. problems and possible solutions in a specific field
>2. proposals for others to replicate
>3. issues (positive or negative) which have emerged from projects
>4. issues you want to raise which you feel have not been discussed yet
>5. issues which are at the centre of an online debate that you would
>like to address offline
>
> "
>
> 1-4 are IMHO relatively easy to evaluate - I would expect from the reviews
> to answer yes or no to them. 5 is a bit tricky as it depends strongly of
> what the reviewer think is "at the centre"  - but I would expect that they
> at least explain in few words here what they think is "at the centre" or
> not :-)
>
>
> 2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak :
>
>> hi,
>>
>> I have some comments as a person from Academia (and not involved in
>> Wikimania process in any way):
>>
>> 1. Short reviews are definitely not helping in addressing the frustration
>> of rejection, yet are quite common in academic peer reviewing, especially
>> for conferences.
>>
>> 2. Double blind peer review (not knowing who is reviewed, and not knowing
>> who reviews) is a standard in Academia, although some perceive it as
>> contributing to lack of responsibility (especially true in competitive
>> journal submissions).
>>
>> 3. Two reviewers per submission is absolutely on par with the conference
>> standards I'm used to. Sometimes there are three, but two is absolutely
>> acceptable (although a third opinion should be used if the two disagree too
>> much).
>>
>> 4. It could be useful to sensitize the reviewers that the main purpose of
>> the review is to help the author to do better next time.
>>
>> 5. All this is volunteer work. We should be, generally, grateful to
>> reviewers (but in the same time grateful to the contributors, too).
>>
>> best,
>>
>> dj
>>
>> On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers 
>> wrote:
>>
>>> What kind of ridiculous process is this? This is all I got:
>>>
>>> ===
>>>
>>> --- REVIEW 1 -
>>> PAPER: 194
>>> TITLE: GLAM+Wikidata
>>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>>>
>>> OVERALL EVALUATION: 8 (Very good)
>>>
>>> --- REVIEW ---
>>> 8
>>>
>>>
>>> --- REVIEW 2 -
>>> PAPER: 194
>>> TITLE: GLAM+Wikidata
>>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>>>
>>> OVERALL EVALUATION: 6 (Rather interesting)
>>>
>>> --- REVIEW ---
>>> 6
>>>
>>> ==
>>>
>>> So only two people reviewed this? Who are these people? Why is this secret? 
>>> Last year I had 5 people reviewing my submission [1].
>>>
>>> Maarten
>>>
>>> [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5
>>>
>>>
>>> Op 3-2-2016 om 23:15 schreef Andy Mabbett:
>>>
>>> I've just received feedback on one of my pitches saying, in part:
>>>
>>> "Bad boy Andy! This is supposed to be an anonymous review process, so
>>> starting your abstract with your own name, is not entirely fair."
>>>
>>> --
>>> Andy Mabbett
>>> @pigsonthewing
>>> http://pigsonthewing.org.uk
>>>
>>>
>>> ___
>>> Wikimania-l mailing 
>>> listWikimania-l@lists.wikimedia.orghttps://lists.wikimedia.org/mailman/listinfo/wikimania-l
>>>
>>>
>>>
>>> ___
>>> Wikimania-l mailing list
>>> Wikimania-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>>>
>>>
>>
>>
>> --
>>
>> __
>> prof. dr hab. Dariusz Jemielniak
>> kierownik katedry Zarządzania Międzynarodowego
>> i grupy badawczej NeRDS
>> Akademia Leona Koźmińskiego
>> http://n wrds.kozminski.edu.pl
>>
>> członek Akademii Młodych Uczonych Polskiej Akademii Nauk
>> członek Komitetu Polityki Naukowej MNiSW
>>
>> Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An
>> 

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Tomasz Ganicz]

It is funny that my submission was rejected on the opposite POV as
according to the reviewer:

"This is a community conference. There are plenty of opportunities for
chapters, WMF, and politicians to get together. Esino Lario is not one of
them."

I don't know why politicians were mentioned by review as the submission was
about education programs of chapters and WMF, but anyway - yes it was
addressed to Wikimedia educators - which of course can meet somewhere else,
not necessarily in Esino Laro.

So - if you address your submission to the Wikimedia community - the issue
is that you can contact community on this list for example, so it is good
reason to reject, but if you address this to the more specific group they
tell you that WIkimania is a meeting for "community" :-)

Taking such statements seriously - you can always say that you can meet
anyone somewhere else to discuss the subject with them.

Also - this kind of  the reviewer statements do not correspond to what was
written on criteria page:

"Please note that your presentation does not need to be for everyone; you
can specify the target you would like to address."




2016-02-04 11:24 GMT+01:00 Federico Leva (Nemo) :

> Tomasz Ganicz, 04/02/2016 10:54:
>
>> 1-4 are IMHO relatively easy to evaluate - I would expect from the
>> reviews to answer yes or no to them. 5 is a bit tricky as it depends
>> strongly of what the reviewer think is "at the centre"  - but I would
>> expect that they at least explain in few words here what they think is
>> "at the centre" or not :-)
>>
>
> Yet, reviews like the one Andy mentioned clearly address this: his review
> was a clear "no" for point 5 as it pointed to (the absence of) a mailing
> list discussion. (I can't check whether that's true.)
>
> Nemo
>
>
> ___
> Wikimania-l mailing list
> Wikimania-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>



-- 
Tomek "Polimerek" Ganicz
http://pl.wikimedia.org/wiki/User:Polimerek
http://www.ganicz.pl/poli/
___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Federico Leva (Nemo)]

Tomasz Ganicz, 04/02/2016 10:54:

1-4 are IMHO relatively easy to evaluate - I would expect from the
reviews to answer yes or no to them. 5 is a bit tricky as it depends
strongly of what the reviewer think is "at the centre"  - but I would
expect that they at least explain in few words here what they think is
"at the centre" or not :-)


Yet, reviews like the one Andy mentioned clearly address this: his 
review was a clear "no" for point 5 as it pointed to (the absence of) a 
mailing list discussion. (I can't check whether that's true.)


Nemo

___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Lodewijk]

Hi all,

quick update about the discussions track: I'm coordinating that together
with a few others, and this will probably be a bit later in time that most
of the other sessions. This is a continuation of the Discussion Room of
2014/2015 , and
focuses on roundtabe discussions of 40-45 minutes each on a specific topic.
That is mostly because there's less preparation required for those
discussions, so we love to have them decided a bit later and have them be
more 'hot and current'. We hope to open improve the descriptions of that
soon though, and open suggestions for that too.

We're currently planning to formally open discussion suggestions on Feb 25
or a bit before. But in the mean time, if you're afraid to forget to submit
it by then, feel free to leave behind ideas here:
https://wikimania2016.wikimedia.org/w/index.php?title=Talk:Discussions=edit=1
(Shani's list of points sounds good in general, although proposals with
less information are also welcome for roundtable discussions)


Best,
Lodewijk


On Thu, Feb 4, 2016 at 9:32 PM, Shani  wrote:

> Following Andrew's comments, here's what I know --
>
> 1 - User Digest is just one session of about 30 minutes. Liaisons pick the
> speakers to that. It's supposed to give a review of the thematic subject -
> GLAM, EDU, etc.
>
> 2 - I'm posting here what I've sent to the Cultural Partners Mailing List -
>
> *​"*I've just updated the GLAM part on the program liaison page on Meta
> ,
> putting there everything from our joint google doc.
>
> Now that the "critical issues" submission part is over, *it's high time
> to submit your suggestions *to all the other aspects of our GLAM track,
> if you haven't done so thus far. This includes suggestions for:
> ** Discussions*
> ** Workshops / Training*
> ** Posters*
> ** Lightning talks*
> * *Anything else we might have forgotten*
>
> Some of you have already contacted me privately about *discussions *and
> *workshops,* so please feel free to update the relevant part on meta.
> Try to keep it in the same format as suggested below, so it's easier to
> follow -
> * Title:
> * Purpose:
> * Target audience:
> * Length:
> * Max number of people (only if there is a limitation on your part):
> * Facilitator(s):
> * any other detail that will help others get a sense of the workshop and
> what you want to achieve.
> Please see an example I posted on behalf of Barbara Fischer -- building
> the GLAM KIT library
> 
> .* "*
>
> In other words*, use the liaison page for now,* till the organizing team
> has the separate pages ready.
> - Follow the format suggest, so it's cohesive and easier to follow.
> - Show your support to proposals, the wiki-way. The organizing team will
> take that into consideration.
> - When the organizing team opens submissions for the remaining parts --
> submit!
>
> Hope that helps,
> Shani.
>
> On Thu, Feb 4, 2016 at 7:39 PM, Marc A. Pelletier 
> wrote:
>
>> On 2016-02-04 3:22 AM, WereSpielChequers wrote:
>>
>>> Hope Montreal manages something a bit better,
>>>
>>
>> I don't know about "better", nor do I think it quite fair to slam the
>> 2016 team either for what was clearly intended to be an attempt to improve
>> the process - even if some of the results appear suboptimal in retrospect.
>>
>> FWIW, the Montreal team is keeping a close eye on the experiments being
>> done by the Italian team - no doubt there will be a valuable set of lessons
>> learned and we may be able to translate some of the things that worked well
>> into improvements to future Wikimanias.
>>
>> As for the programme selection, we are gunning for a process that splits
>> about 30% invited, 40% community CFP, and 30% unconference-style, with the
>> selection process for the CFP being very close to past years (i.e.: public
>> review on-wiki).  We also don't intend to make a distinction between
>> submissions by Foundation staff and the other community members, though we
>> expect that many presentations that would have been proposals by staff will
>> end up being invited directly by the programming committee leaving more
>> "slots" available to the CFP.
>>
>> -- Marc
>>
>>
>>
>> ___
>> Wikimania-l mailing list
>> Wikimania-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>>
>
>
> ___
> Wikimania-l mailing list
> Wikimania-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>
>
___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Lydia Pintscher]

On Thu, Feb 4, 2016 at 2:59 PM Marc A. Pelletier  wrote:

> On 2016-02-04 3:22 AM, WereSpielChequers wrote:
> > But when you get the same presentation being marked as 5 and 8
>
> I got a 4 and a 8; the former noting in the comment that it was not
> clear that I knew the subject.  Subject being external tools to the
> project.  Like found on Tool Labs.
>
> I'm not sure that review process was entirely successful.
>

I got a very similar comment assuming I knew little about Wikidata. As its
product manager...


Cheers
Lydia
-- 
Lydia Pintscher - http://about.me/lydia.pintscher
Product Manager for Wikidata

Wikimedia Deutschland e.V.
Tempelhofer Ufer 23-24
10963 Berlin
www.wikimedia.de

Wikimedia Deutschland - Gesellschaft zur Förderung Freien Wissens e. V.

Eingetragen im Vereinsregister des Amtsgerichts Berlin-Charlottenburg unter
der Nummer 23855 Nz. Als gemeinnützig anerkannt durch das Finanzamt für
Körperschaften I Berlin, Steuernummer 27/029/42207.
___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Fæ]

On 4 February 2016 at 14:50, Lydia Pintscher
 wrote:
> I got a very similar comment assuming I knew little about Wikidata. As its
> product manager...

Anyone who has been burdened with doing these sorts of reviews will
feel some sympathy for those giving the feedback. It is easy to upset
a lot of people if the process is not well thought out. Where there
are marking discrepancies, the workflow should mean it goes to another
independent reviewer and there is a meeting (like 2 minutes in a
Hangout discussion) where there is final agreement on the rating/mark
*and* the feedback that should be given.

Even without discrepancies in marks, feedback needs to be positive and
supportive, this is all volunteers giving their time after all, not
postgrads getting critical essay feedback. That means the workflow
also needs to include regular checks and team meetings to talk about
how to best ensure marks and feedback remains consistent, even when
the experience and viewpoints of the reviewers may be highly varied.

Lots of lessons to be summarized for later, and probably a need to
consider whether now is a good time put up your hands and formally
admit to problems in consistency. Asking submitters to give their
feedback and suggestions on-wiki, even if is too late to change any
decision, was a good response.

Fae
-- 
fae...@gmail.com https://commons.wikimedia.org/wiki/User:Fae

___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[WereSpielChequers]

That seems a sensible system, I can see how a 6 and a 7 were enough to reject 
one of my proposals, though I'd have appreciated a bit more feedback as to why.

But how strong did the divergence have to be if 5 and 8 didn't count as a 
"strong divergence"?

WereSpielChequers




> On 4 Feb 2016, at 11:55, Sebastian Wallroth  wrote:
> 
> Hi Tomasz,
> 
> this is what actually happend. Please refer to
> https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation
> 
> 
> : In order to achieve the greatest possible neutrality, the submissions
> will
> : be evaluated online using a *double-blind peer-review process*. This
> : means that two evaluators will review the submission without knowing
> : the name of its author. *If there are strong divergences among the two
> : evaluations, at least one other review will be made.*
> 
> Kind regards,
> Sebastian
> 
>> Am 04.02.2016 um 11:31 schrieb Tomasz Ganicz:
>> Also - normally in Academia - if there are two strongly opposite
>> reviews (one very positive, one very negative) a typical procedure is
>> to send the submission to the third one.
>> 
>> 2016-02-04 10:54 GMT+01:00 Tomasz Ganicz > >:
>> 
>>Well I think that double blind peer review hardly make sense here,
>>from the reviewer POV, as we are in fact small community and it
>>is  easy to guess who was a submiter in most cases. For example -
>>if there is a submission about project X in country Y, which was
>>funded by WMF grant - it is very easy to find out who was grantee
>>and it is rather obvious that that person is a submitter :-)
>> 
>>Also  judging from the several reviewers comments which I saw
>>already - they did not follow the very vague criteria which was
>>posted here:
>> 
>>
>> https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Evaluation
>> 
>>Normally - at least in Academia - reviewers are forced directly
>>(by the review form) to address their opinion in relation to the
>>criteria. The criteria were:
>> 
>>"
>> 
>> 1. problems and possible solutions in a specific field
>> 2. proposals for others to replicate
>> 3. issues (positive or negative) which have emerged from projects
>> 4. issues you want to raise which you feel have not been
>>discussed yet
>> 5. issues which are at the centre of an online debate that you
>>would like to address offline
>> 
>>"
>> 
>>1-4 are IMHO relatively easy to evaluate - I would expect from the
>>reviews to answer yes or no to them. 5 is a bit tricky as it
>>depends strongly of what the reviewer think is "at the centre"  -
>>but I would expect that they at least explain in few words here
>>what they think is "at the centre" or not :-)
>> 
>> 
>> 
>>2016-02-04 0:22 GMT+01:00 Dariusz Jemielniak >>:
>> 
>>hi,
>> 
>>I have some comments as a person from Academia (and not
>>involved in Wikimania process in any way):
>> 
>>1. Short reviews are definitely not helping in addressing the
>>frustration of rejection, yet are quite common in academic
>>peer reviewing, especially for conferences. 
>> 
>>2. Double blind peer review (not knowing who is reviewed, and
>>not knowing who reviews) is a standard in Academia, although
>>some perceive it as contributing to lack of responsibility
>>(especially true in competitive journal submissions).
>> 
>>3. Two reviewers per submission is absolutely on par with the
>>conference standards I'm used to. Sometimes there are three,
>>but two is absolutely acceptable (although a third opinion
>>should be used if the two disagree too much). 
>> 
>>4. It could be useful to sensitize the reviewers that the main
>>purpose of the review is to help the author to do better next
>>time. 
>> 
>>5. All this is volunteer work. We should be, generally,
>>grateful to reviewers (but in the same time grateful to the
>>contributors, too). 
>> 
>>best,
>> 
>>dj
>> 
>>On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers
>>> wrote:
>> 
>>What kind of ridiculous process is this? This is all I got:
>> 
>>===
>> 
>>--- REVIEW 1 -
>>PAPER: 194
>>TITLE: GLAM+Wikidata
>>AUTHORS: Sandra Fauconnier and Maarten Dammers
>> 
>>OVERALL EVALUATION: 8 (Very good)
>> 
>>--- REVIEW ---
>>8
>> 
>> 
>>--- REVIEW 2 -
>>PAPER: 194
>>TITLE: GLAM+Wikidata
>>AUTHORS: Sandra Fauconnier and Maarten Dammers
>> 
>>   

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Andrew Lih]

One thing that would help make sense of where we are now: remind us how the
overall structure of the conference is going to include not just the 42
“critical issues” sessions picked out of EasyChair, but also ones via other
processes. Make it clear, repeat it constantly, and give links to people to
understand it.

Right now, I cannot figure out the proportion, appropriateness or overall
relationship of user digest presentations, critical issues or discussions,
as laid out here: https://wikimania2016.wikimedia.org/wiki/Submissions

For example, consider the high profile SXSW conference. They represent the
content breakdown like this:
40% programming committee
30% public votes
30% staff

This tries to assure folks that good content will get recognized through
one of three different processes.

Since this year’s Wikimania process is so new, there’s a lot of confusion
on how to slot in other ideas outside of the formal EasyChair submissions.
To wit, on the Submissions page of Wikimania 2016:

- “User digest presentations" - When the page says “Contact the Thematic
Liaison,” the user is almost always at least two clicks away from finding a
way to contact that person. Even worse, for many users, clicking on their
name sends you to a confusing page: “
https://wikimania2016.wikimedia.org/wiki/Special:CentralAuth/Username“ What
is the average user supposed to do with that?

- “Discussions” page is a red link. There is not even a simple description
of what this is. Same thing with “Community Village”

- Even a brief paragraph or a diagram showing the 10,000 foot/3048 meter
view of the overall plan would be welcome on the submissions page. Right
now there is no prose, only six big categories. This requires a lot of
haphazard clicking and piecing together of the conference narrative. The
only thing I found useful to describe the overall programming strategy is
in the what is “NOT accepted” list in the critical issues guidelines, as
described in this subsection:
https://wikimania2016.wikimedia.org/wiki/Critical_issues_presentations#Topic


I don’t mean to pile-on the Wikimania 2016 team, as I know how arduous it
is to do this conference. I hope you’ll see this as not just griping, but
constructive feedback on how to make the site and process better for users.

Thanks.

-Andrew Lih
Associate professor of journalism, American University
Email: and...@andrewlih.com
WEB: http://www.andrewlih.com
BOOK: The Wikipedia Revolution: http://www.wikipediarevolution.com
PROJECT: Wiki Makes Video
http://en.wikipedia.org/wiki/Wikipedia:WikiProject_Wiki_Makes_Video

On Thu, Feb 4, 2016 at 10:11 AM, Fæ  wrote:

> On 4 February 2016 at 14:50, Lydia Pintscher
>  wrote:
> > I got a very similar comment assuming I knew little about Wikidata. As
> its
> > product manager...
>
> Anyone who has been burdened with doing these sorts of reviews will
> feel some sympathy for those giving the feedback. It is easy to upset
> a lot of people if the process is not well thought out. Where there
> are marking discrepancies, the workflow should mean it goes to another
> independent reviewer and there is a meeting (like 2 minutes in a
> Hangout discussion) where there is final agreement on the rating/mark
> *and* the feedback that should be given.
>
> Even without discrepancies in marks, feedback needs to be positive and
> supportive, this is all volunteers giving their time after all, not
> postgrads getting critical essay feedback. That means the workflow
> also needs to include regular checks and team meetings to talk about
> how to best ensure marks and feedback remains consistent, even when
> the experience and viewpoints of the reviewers may be highly varied.
>
> Lots of lessons to be summarized for later, and probably a need to
> consider whether now is a good time put up your hands and formally
> admit to problems in consistency. Asking submitters to give their
> feedback and suggestions on-wiki, even if is too late to change any
> decision, was a good response.
>
> Fae
> --
> fae...@gmail.com https://commons.wikimedia.org/wiki/User:Fae
>
> ___
> Wikimania-l mailing list
> Wikimania-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>
___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Federico Leva (Nemo)]

Lodewijk, 03/02/2016 15:49:

lots of people were quite surprised by the outcomes of the review.


This sounds like a good thing! If reviews matched expectations, there 
would be no point in having a review. :)


Nemo

___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Marc A. Pelletier]

On 2016-02-04 3:22 AM, WereSpielChequers wrote:

Hope Montreal manages something a bit better,


I don't know about "better", nor do I think it quite fair to slam the 
2016 team either for what was clearly intended to be an attempt to 
improve the process - even if some of the results appear suboptimal in 
retrospect.


FWIW, the Montreal team is keeping a close eye on the experiments being 
done by the Italian team - no doubt there will be a valuable set of 
lessons learned and we may be able to translate some of the things that 
worked well into improvements to future Wikimanias.


As for the programme selection, we are gunning for a process that splits 
about 30% invited, 40% community CFP, and 30% unconference-style, with 
the selection process for the CFP being very close to past years (i.e.: 
public review on-wiki).  We also don't intend to make a distinction 
between submissions by Foundation staff and the other community members, 
though we expect that many presentations that would have been proposals 
by staff will end up being invited directly by the programming committee 
leaving more "slots" available to the CFP.


-- Marc


___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[Shani]

Following Andrew's comments, here's what I know --

1 - User Digest is just one session of about 30 minutes. Liaisons pick the
speakers to that. It's supposed to give a review of the thematic subject -
GLAM, EDU, etc.

2 - I'm posting here what I've sent to the Cultural Partners Mailing List -

*​"*I've just updated the GLAM part on the program liaison page on Meta
,
putting there everything from our joint google doc.

Now that the "critical issues" submission part is over, *it's high time to
submit your suggestions *to all the other aspects of our GLAM track, if you
haven't done so thus far. This includes suggestions for:
** Discussions*
** Workshops / Training*
** Posters*
** Lightning talks*
* *Anything else we might have forgotten*

Some of you have already contacted me privately about *discussions *and
*workshops,* so please feel free to update the relevant part on meta.
Try to keep it in the same format as suggested below, so it's easier to
follow -
* Title:
* Purpose:
* Target audience:
* Length:
* Max number of people (only if there is a limitation on your part):
* Facilitator(s):
* any other detail that will help others get a sense of the workshop and
what you want to achieve.
Please see an example I posted on behalf of Barbara Fischer -- building the
GLAM KIT library

.* "*

In other words*, use the liaison page for now,* till the organizing team
has the separate pages ready.
- Follow the format suggest, so it's cohesive and easier to follow.
- Show your support to proposals, the wiki-way. The organizing team will
take that into consideration.
- When the organizing team opens submissions for the remaining parts --
submit!

Hope that helps,
Shani.

On Thu, Feb 4, 2016 at 7:39 PM, Marc A. Pelletier  wrote:

> On 2016-02-04 3:22 AM, WereSpielChequers wrote:
>
>> Hope Montreal manages something a bit better,
>>
>
> I don't know about "better", nor do I think it quite fair to slam the 2016
> team either for what was clearly intended to be an attempt to improve the
> process - even if some of the results appear suboptimal in retrospect.
>
> FWIW, the Montreal team is keeping a close eye on the experiments being
> done by the Italian team - no doubt there will be a valuable set of lessons
> learned and we may be able to translate some of the things that worked well
> into improvements to future Wikimanias.
>
> As for the programme selection, we are gunning for a process that splits
> about 30% invited, 40% community CFP, and 30% unconference-style, with the
> selection process for the CFP being very close to past years (i.e.: public
> review on-wiki).  We also don't intend to make a distinction between
> submissions by Foundation staff and the other community members, though we
> expect that many presentations that would have been proposals by staff will
> end up being invited directly by the programming committee leaving more
> "slots" available to the CFP.
>
> -- Marc
>
>
>
> ___
> Wikimania-l mailing list
> Wikimania-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>
___
Wikimania-l mailing list
Wikimania-l@lists.wikimedia.org
https://lists.wikimedia.org/mailman/listinfo/wikimania-l

Re: [Wikimania-l] Wikimania 2016 - Notifications of acceptance and rejection for "Critical issues" presentations

[WereSpielChequers]

Two reviews per submission might work if we had a clear set of criteria that 
the reviewers were following and sufficient training of the reviewers that they 
were broadly consistent in their marking. But when you get the same 
presentation being marked as 5 and 8, as one of mine was then the suspicion is 
that the assessors are not working to the same criteria as each other. That 
wouldn't matter so much if they were all assessing all submissions, except that 
an assessor who varied between 0 and ten points would have far more influence 
than assessors who usually voted 6, 7 or 8. But having that level of 
inconsistency and only two reviews per submission makes the process a lottery 
that depends on who the two reviewers are for your submission.

As for the content of the reviews, I don't consider that either "5 (average)" 
or "6 (rather interesting) tell me anything as to why my submissions were 
rejected.

The other two reviews at least managed one or two lines. One of them even 
stretched to two sentences.

Hope Montreal manages something a bit better, I'm sure either Manilla or Perth 
would have done.


WereSpielChequers


> On 3 Feb 2016, at 23:22, Dariusz Jemielniak  wrote:
> 
> hi,
> 
> I have some comments as a person from Academia (and not involved in Wikimania 
> process in any way):
> 
> 1. Short reviews are definitely not helping in addressing the frustration of 
> rejection, yet are quite common in academic peer reviewing, especially for 
> conferences. 
> 
> 2. Double blind peer review (not knowing who is reviewed, and not knowing who 
> reviews) is a standard in Academia, although some perceive it as contributing 
> to lack of responsibility (especially true in competitive journal 
> submissions).
> 
> 3. Two reviewers per submission is absolutely on par with the conference 
> standards I'm used to. Sometimes there are three, but two is absolutely 
> acceptable (although a third opinion should be used if the two disagree too 
> much). 
> 
> 4. It could be useful to sensitize the reviewers that the main purpose of the 
> review is to help the author to do better next time. 
> 
> 5. All this is volunteer work. We should be, generally, grateful to reviewers 
> (but in the same time grateful to the contributors, too). 
> 
> best,
> 
> dj
> 
>> On Wed, Feb 3, 2016 at 5:26 PM, Maarten Dammers  wrote:
>> What kind of ridiculous process is this? This is all I got:
>> 
>> ===
>> 
>> --- REVIEW 1 -
>> PAPER: 194
>> TITLE: GLAM+Wikidata
>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>> 
>> OVERALL EVALUATION: 8 (Very good)
>> 
>> --- REVIEW ---
>> 8
>> 
>> 
>> --- REVIEW 2 -
>> PAPER: 194
>> TITLE: GLAM+Wikidata
>> AUTHORS: Sandra Fauconnier and Maarten Dammers
>> 
>> OVERALL EVALUATION: 6 (Rather interesting)
>> 
>> --- REVIEW ---
>> 6
>> 
>> ==
>> 
>> So only two people reviewed this? Who are these people? Why is this secret? 
>> Last year I had 5 people reviewing my submission [1].
>> 
>> Maarten
>> 
>> [1] https://wikimania2015.wikimedia.org/wiki/Submission_review/5
>> 
>> Op 3-2-2016 om 23:15 schreef Andy Mabbett:
>>> I've just received feedback on one of my pitches saying, in part:
>>> 
>>> "Bad boy Andy! This is supposed to be an anonymous review process, so 
>>> starting your abstract with your own name, is not entirely fair."
>>> 
>>> -- 
>>> Andy Mabbett
>>> @pigsonthewing
>>> http://pigsonthewing.org.uk
>>> 
>>> 
>>> 
>>> ___
>>> Wikimania-l mailing list
>>> Wikimania-l@lists.wikimedia.org
>>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
>> 
>> 
>> ___
>> Wikimania-l mailing list
>> Wikimania-l@lists.wikimedia.org
>> https://lists.wikimedia.org/mailman/listinfo/wikimania-l
> 
> 
> 
> -- 
> 
> __
> prof. dr hab. Dariusz Jemielniak
> kierownik katedry Zarządzania Międzynarodowego
> i grupy badawczej NeRDS
> Akademia Leona Koźmińskiego
> http://nwrds.kozminski.edu.pl 
> 
> członek Akademii Młodych Uczonych Polskiej Akademii Nauk
> członek Komitetu Polityki Naukowej MNiSW
> 
> Wyszła pierwsza na świecie etnografia Wikipedii "Common Knowledge? An 
> Ethnography of Wikipedia" (2014, Stanford University Press) mojego autorstwa 
> http://www.sup.org/book.cgi?id=24010
> 
> Recenzje
> Forbes: http://www.forbes.com/fdc/welcome_mjx.shtml
> Pacific Standard: 
> http://www.psmag.com/navigation/books-and-culture/killed-wikipedia-93777/
> Motherboard: http://motherboard.vice.com/read/an-ethnography-of-wikipedia
> The Wikipedian: 
> http://thewikipedian.net/2014/10/10/dariusz-jemielniak-common-knowledge
> ___
> Wikimania-l mailing list
> Wikimania-l@lists.wikimedia.org
> https://lists.wikimedia.org/mailman/listinfo/wikimania-l