Since the Hutter prize was expanded 10x to 5000 € per 1% improvement in 2019, the historical rate of improvement has been 0.5% per year. Meanwhile the top 5 entries on LTCB beat the Hutter prize, with the best at 6% smaller.
I think that anyone with the technical skills to submit a winning entry is probably not doing it for the prize money. I think if the goal is to understand language modeling, or even just to find the Kolmogorov complexity of enwik9, then we should at least relax the hardware limits to something comparable to the value of the research. I probably invested $1M of my time in developing the PAQ algorithm as a hobby over a decade, which I made back in the next 6 years before I retired. On Thu, Jul 6, 2023, 1:29 PM James Bowery <[email protected]> wrote: > > > On Thu, Jul 6, 2023 at 12:00 PM Matt Mahoney <[email protected]> > wrote: > >> On Thu, Jul 6, 2023, 2:09 AM Rob Freeman <[email protected]> >> wrote: >> >>> >>> Did the Hutter Prize move the field? Well, I was drawn to it as a rare >>> data based benchmark. >>> >> >> Not much. I disagreed with Hutter on the contest rules, but he was >> funding the prize. (I once applied for an NSF grant but it was rejected >> like 90% of applications). My large text benchmark has no limits on CPU >> time or memory, but I understand the need for these when there is prize >> money. >> > > There are really two aspects to the "failure" of the Hutter Prize to "move > the field". > > The first and most important aspect is that one must compare the levelized > cost of $100/month of the Hutter Prize since its inception to that invested > in "the field". Arguments over the resource limits imposed, and even the > dataset to be compressed are secondary. This brings up the second aspect > of its "failure: It remains unrecognized that there is virtually no risk > to investment in the Hutter Prize purse, and there are rigorous grounds for > believing it to be the best we can do for addressing *precisely* the > kinds of language model failures we're seeing exposed in the LLMs -- not > just enormous costs but also their inability to factor out various kinds of > internal inconsistencies, "noise" and outright lies in the source data. > > One can argue, as you have, that the resource limits are too strict to > move the needle because it is just *too hard*. One must recognize that > money is only paid out *only *as one approaches the *hard* asymptotic > limit in the Kolmogorov Complexity of the dataset. Why *not* fund it at > a level of a billion dollars? There's no downside and an *enormous* > upside! > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T42db51de471cbcb9-M62df78dea340bae690ad896f> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T42db51de471cbcb9-M00cb7811d62b0be50244014b Delivery options: https://agi.topicbox.com/groups/agi/subscription
