On Thu, Jul 6, 2023 at 2:03 PM Matt Mahoney <[email protected]> wrote:
> Since the Hutter prize was expanded 10x to 5000 € per 1% improvement in > 2019, the historical rate of improvement has been 0.5% per year. Meanwhile > the top 5 entries on LTCB beat the Hutter prize, with the best at 6% > smaller. > > I think that anyone with the technical skills to submit a winning entry is > probably not doing it for the prize money. > There is at least one company with an algorithm demonstrating 10x data efficiency improvement over transformers and 10x improvement in compute resources that looked at the Hutter Prize but decided not to pursue it because the prize purse wasn't large enough given the fact that the industry doesn't recognize the benchmark as being as meaningful as less principled metrics like perplexity. > I think if the goal is to understand language modeling, or even just to > find the Kolmogorov complexity of enwik9, then we should at least relax the > hardware limits to something comparable to the value of the research. > They wouldn't pursue it that way either -- particularly since their bragging rights have to do with decreased resource utilization. It certainly would make sense to have an enwik9-based prize without resource constraints since there is such a huge "market gap" between the Hutter Prize and the resource hogs being thrown at the LLMs. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T42db51de471cbcb9-M197c85c4fd77f469d53fd321 Delivery options: https://agi.topicbox.com/groups/agi/subscription
