The Large Text Benchmark and Hutter prize test language modeling algorithms, not language models. An actual language model wouldn't be trained on just 1 GB of Wikipedia from 2006. But what we learned from this is that neural networks is the way to go, specifically transformers running on GPUs.
On Tue, Jul 23, 2024, 3:10 PM James Bowery <[email protected]> wrote: > I directed the question at you because you are likely to understand how > different training and inference are since you said you "pay my bills by > training" -- so far from levelling a criticism at you I was hoping you had > some insight into the failure of the industry to use training benchmarks as > opposed to inference benchmarks. > > Are you saying you don't see the connection between training and > compression? > > On Mon, Jul 22, 2024 at 8:08 PM Aaron Hosford <[email protected]> wrote: > >> Sorry, I'm not sure what you're saying. It's not clear to me if this is >> intended as a criticism of me, or of someone else. Also, I lack the context >> to draw the connection between what I've said and the topic of >> compression/decompression, I think. >> >> On Mon, Jul 22, 2024 at 5:17 PM James Bowery <[email protected]> wrote: >> >>> >>> >>> On Mon, Jul 22, 2024 at 4:12 PM Aaron Hosford <[email protected]> >>> wrote: >>> >>>> ... >>>> >>>> I spend a lot of time with LLMs these days, since I pay my bills by >>>> training them.... >>>> >>> >>> Maybe you could explain why it is that people who get their hands dirty >>> training LLMs, and are therefore acutely aware of the profound difference >>> between training and inference (if for no other reason than that training >>> takes orders of magnitude more resources), seem to think that these >>> benchmark tests should be on the inference side of things whereas the >>> Hutter Prize has, *since 2006*, been on the training *and* inference >>> side of things, because a winner must both train (compress) and infer >>> (decompress). >>> >>> Are the "AI experts" really as oblivious to the obvious as they appear >>> and if so *why*? >>> >> *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T6510028eea311a76-M3f44388f09277d0c433374da> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T6510028eea311a76-Mb81011d0bfa13655b772ecae Delivery options: https://agi.topicbox.com/groups/agi/subscription
