Here are a few of the challenges: - How many people solve a problem depends on the context of the problem. If it's problem D, then even if it isn't that hard, it isn't going to get looked at by fewer people than problem A. I have no idea how to deal with this. - It's hard to rank a problem attacked by the top 500 against a problem attacked by the top 10,000. I think this is the dominant challenge.
*Here's a way to construct a partial ordering:* OK, here's an idea. I was going to say it's I think it's very, very difficult to rank a problem from a round with the 500 best people against a problem from a round with everyone who passed the Qualification Round. But there's no law that says you have to look at *everybody* who solved a problem. - Compare problems from *adjacent rounds* against each other, and only use people who submitted in both rounds. So for example, you can compare last year's Round 3 problems (500 participants) against last year's Round 2 problems (3000 participants). You can rank all eight problems by how many of those 500 (well, the 375 on the scoreboard) solved them. We only compare adjacent rounds because that lets you say "According to the people in Round 2 and Round 3, how tough were Round 2's problems compared to Round 3's?" - Compare problems from some limited range of *years* (I think some meaning is lost comparing 2008 against 2017) against each other, only within rounds. Presumably 2017's Round 2 participants were similarly skilled to 2016's Round 2 participants, so you can compare those problems against each other pretty well. When you're comparing Qualification Round (and maybe R1) problems against each other, you should probably normalize by the number of contestants, which isn't fixed. - I haven't thought through what happens after you have your partial ordering. Maybe you break ties based on how many people solved the whole problem.Is it possible to have an inconsistent partial ordering? Maybe, I'm not sure. *Another idea: Elo* Or something better, which I bet somebody from this list can come up with. Treat each contestant as a chess player, playing matches against problems. Each contestant and each problem ends up with a rating of some sort, and you rank the problems by rating. On Fri, Apr 27, 2018 at 2:34 AM Paul Smith <[email protected]> wrote: > Sorry, I mean the total number of people who attempted any problem in that > problem set. I think the scoreboard doesn't show you records of people who > logged in, looked at the problem, then logged out and went back to sleep, > but it will record people who uploaded even a single small input. So if > you know that a competition round had 5000 active participants, and 3000 > submitted a correct small input, that problem scores 60% for difficulty, > and if another round had 20000 active participants, 10000 of which > submitted a good input, that scores 50% for difficulty. > > You might need to apply some kind of adjustment for rounds though. A > Round 3 problem that 22 out of 25 participants solved is still not going to > be easy! > > On Fri, 27 Apr 2018 at 09:26 Paul Smith <[email protected]> wrote: > >> Try dividing by the number of people who solved anything at all that year. >> >> On Thu, 26 Apr 2018 at 22:23 Anil M <[email protected]> wrote: >> >>> I am solving problem on codejam over the years with no definite >>> pattern. I wondered all the problems are scattered and not easy to find >>> the easiest problem. So I crawled all the problems and got the stats for >>> each problem and put them in a database. Now I want to order them based on >>> easiness. I choose to order them based on number of people solved the small >>> input in descending order, this worked pretty well, all easy problems at >>> top, but with one issue, In the years 2008,2009 number of people who >>> attempted the problems is less, so a problem in 2008 solved by 3000 people >>> could be easier than a problem in 2017 solved by 10000 people. Alternative >>> approach I thought was asking coders to order, say to put a up and down >>> button on each problem, if they press up button means they think it is >>> easier than it appears in the list of problems, down button means >>> otherwise. What do you think is the best strategy to order all the >>> problems based on easiness/difficulty, so I can always solve easiest >>> problem I didn't solve yet? (You can find all problems at >>> www.manchik.co.uk/list to get an idea what I'm trying to do) >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "Google Code Jam" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To post to this group, send email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/google-code/243f5d0d-b3b8-4a10-bba1-3a48bcf8e6b9%40googlegroups.com >>> . >>> For more options, visit https://groups.google.com/d/optout. >>> >> -- > You received this message because you are subscribed to the Google Groups > "Google Code Jam" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To post to this group, send email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/google-code/CAJej63L0LTYKNm%3D_BfCWRVT78G7hfVK227EaHf5FeO3AWZwT%2BQ%40mail.gmail.com > <https://groups.google.com/d/msgid/google-code/CAJej63L0LTYKNm%3D_BfCWRVT78G7hfVK227EaHf5FeO3AWZwT%2BQ%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > For more options, visit https://groups.google.com/d/optout. > -- You received this message because you are subscribed to the Google Groups "Google Code Jam" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To post to this group, send email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/google-code/CAHaiWHPJFuHmZneKQAB2%3D8L%2BCvr1o%2BkBBdh%3D-Q9Hp5duBFrJvA%40mail.gmail.com. For more options, visit https://groups.google.com/d/optout.
