Hi, > On 29 Sep 2016, at 08:48, Christofer Dutz <christofer.d...@c-ware.de> wrote: > I just wanted to also take the opportunity to give some feedback on the > modified review process: > > > 1. Seeing that I would have to do 30000 decisions sort of turned me off right > away (It's sort of ... "yeah let me help" and then getting a huge pile of > work dumped on my desk) > > 2. With that huge amount of possible work I could see only little progress > for quite some time put into it) ... 30000 decisions would require reading of > 60000 applications. If I assume 30 seconds per application that's about 500 > hours which is about 20 days without doing anything else. I sort of quit at > about 400 decisions. > > 3. I noticed for myself that at first you start reading the applications > carefully but that accuracy goes down very fast as soon as you get a lot of > the talks you reviewed earlier ... unfortunately even if you only think you > read it before. I noticed me not reading some similar looking applications > and voting for one thinking it's the other. Don't know if this is desirable. > > I liked the simple interface however. So how about dropping the Deathmatch > approach and just displaying one application, and let the user select how > much he likes it (ok ... this is just the way the old version worked, but as > I said, I liked the UI ... just clicking once) ... eventually the user could > also add tags to the application and suggest tracks.
I share this as well: given the large amount of proposals, the decision making for all permutations is simply too much. Also, due to the relatively small amount of reviewers and the “real” randomisation, I think there’s a large bias in the final decisions: I’ve come across the same “match” several times, which implies that the one talk that lost the battle has a negative bias. I’ve tried to do as many “battles” as possible, but got only up to about 1000 before I was fed up and no longer could spend time on it due to other obligations. I’m not sure if I’ve seen all proposals (probably not), which is a pity, in my opinion... -- Met vriendelijke groeten | Kind regards Jan Willem Janssen | Software Architect +31 631 765 814 My world is something with Amdatu and Apache Luminis Technologies Churchillplein 1 7314 BZ Apeldoorn +31 88 586 46 00 https://www.luminis.eu KvK (CoC) 09 16 28 93 BTW (VAT) NL8170.94.441.B.01
signature.asc
Description: Message signed with OpenPGP using GPGMail
