James, have you seen Poggio's attempt to argue that these overparametrized models are actually OK in terms of learning theory?
https://dspace.mit.edu/handle/1721.1/124343 The basic argument seems to be: -- In a space of overparametrized models, stability under subsampling (e.g. leave-one-out accuracy as he describes) is a proxy for minimizing error -- So if it's easier/faster to find overparameterized models than compact models, this can still be a route to maximally accurate models so long as one uses subsampling to estimate accuracy His statistical theory is openly hand-wavy but the conceptual argument is clear... In terms of algorithmic information theory ish considerations, he is sorta implicitly assuming that the crypticity (difficulty of discovering/learning) overparametrized models is less than the crypticity of compact models, which is indeed the experience of the ML and NLP worlds so far I do not think this is the whole story by any means, but it's a non-stupid and relevant (if not actually original) point... ben On Wed, Jul 1, 2020 at 9:08 AM James Bowery <jabow...@gmail.com> wrote: > > There seems to be hysteria against algorithmic information theory in language > modeling. > > OpenAI boasts a 175 BILLION parameter model. > > Now Google boasts a 600 BILLION parameter model. > > https://youtu.be/1VdEw_mGjFk > > Now, I wouldn't call this "anti-AIT" if it weren't for the fact that these > papers don't even attempt to estimate the actual information content of these > parameters. Instead, they seem to take _pride_ in the obviously-inflated > "parameter count". > > > > > > > Artificial General Intelligence List / AGI / see discussions + participants + > delivery options Permalink -- Ben Goertzel, PhD http://goertzel.org “The only people for me are the mad ones, the ones who are mad to live, mad to talk, mad to be saved, desirous of everything at the same time, the ones who never yawn or say a commonplace thing, but burn, burn, burn like fabulous yellow roman candles exploding like spiders across the stars.” -- Jack Kerouac ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T100f708e32ae7327-M38b127dbe701668060df64e3 Delivery options: https://agi.topicbox.com/groups/agi/subscription
