I'm not sure how one would go the next step from a random-speech-generating network like that.
We do want the speech to mean something. My new approach is to incorporate semantics into a rule engine right from the start. On Sun, 17 Feb 2019 at 02:09, Ben Goertzel <[email protected]> wrote: > Rob, > > These deep NNs certainly are not linear models, and they do capture a > bunch of syntactic phenomena fairly subtly, see e.g. > > https://arxiv.org/abs/1901.05287 > > "I assess the extent to which the recently introduced BERT model > captures English syntactic phenomena, using (1) naturally-occurring > subject-verb agreement stimuli; (2) "coloreless green ideas" > subject-verb agreement stimuli, in which content words in natural > sentences are randomly replaced with words sharing the same > part-of-speech and inflection; and (3) manually crafted stimuli for > subject-verb agreement and reflexive anaphora phenomena. The BERT > model performs remarkably well on all cases." > > This paper shows some dependency trees implicit in transformer networks, > > http://aclweb.org/anthology/W18-5431 > > This stuff is not AGI and does not extract deep semantics nor do > symbol grounding etc. For sure it has many limitations. Bu it's > also not so trivial as you're suggesting IMO... > > -- Ben G > > On Sun, Feb 17, 2019 at 8:42 AM Rob Freeman <[email protected]> > wrote: > > > > On the substance, here's what I wrote elsewhere in response to someone's > comment that it is an "important step": > > > > Important step? I don't see it. Bengio's NLM? Yeah, good, we need > distributed representation. That was an advance. but it was always a linear > model without a sensible way of folding in context. Now they try to fold in > a bit of context by bolting on another layer to spotlight other parts of > the sequence ad-hoc? > > > > I don't see any theoretical cohesiveness, any actual theory let alone > novelty of theory. > > > > What is the underlying model for language here? In particular what is > the underlying model for how words combine to create meaning? How do parts > of a sequence combine to become a whole, incorporating the whole context? > Linear combination with a bolt-on spotlight? > > > > I think all this ad-hoc tinkering will be thrown away when we figure out > a principled way to combine words which incorporates context inherently. > But nobody is even attempting that. They are just tinkering. Limited to > tinkering with linear models, because nothing else can be "learned". > > > > On Sun, Feb 17, 2019 at 1:05 PM Ben Goertzel <[email protected]> wrote: > >> > >> Hmmm... > >> > >> About this "OpenAI keeping their language model secret" thing... > >> > >> I mean -- clearly, keeping their language model secret is a pure PR > >> stunt... Their > >> algorithm is described in an online paper... and their model was > >> trained on Reddit text ... so anyone else with a bunch of $$ (for > >> machine-time and data-preprocessing hacking) can download Reddit > >> (complete Reddit archives are available as a torrent) and train a > >> language model similar or better > >> than OpenAI's ... > >> > >> That said, their language model is a moderate improvement on the BERT > >> model released by Google last year. This is good AI work. There is > >> no understanding of semantics and no grounding of symbols in > >> experience/world here, but still, it's pretty f**king cool to see what > >> an awesome job of text generation can be done by these pure > >> surface-level-pattern-recognition methods.... > >> > >> Honestly a lot of folks in the deep-NN/NLP space (including our own > >> SingularityNET St. Petersburg team) have been talking about applying > >> BERT-ish attention networks (with more comprehensive network > >> architectures) in similar ways... but there are always so many > >> different things to work on, and OpenAI should be congratulated for > >> making these particular architecture tweaks and demonstrating them > >> first... but not for the PR stunt of keeping their model secret... > >> > >> Although perhaps they should be congratulated for revealing so clearly > >> the limitations of the "open-ness" in their name "Open AI." I mean, > >> we all know there are some cases where keeping something secret may be > >> the most ethical choice ... but the fact that they're willing to take > >> this step simply for a short-term one-news-cycle PR boost, indicates > >> that open-ness may not be such an important value to them after all... > >> > >> -- > >> Ben Goertzel, PhD > >> http://goertzel.org > >> > >> "Listen: This world is the lunatic's sphere, / Don't always agree > >> it's real. / Even with my feet upon it / And the postman knowing my > >> door / My address is somewhere else." -- Hafiz > > > > Artificial General Intelligence List / AGI / see discussions + > participants + delivery options Permalink > > -- > Ben Goertzel, PhD > http://goertzel.org > > "Listen: This world is the lunatic's sphere, / Don't always agree > it's real. / Even with my feet upon it / And the postman knowing my > door / My address is somewhere else." -- Hafiz -- Stefan Reich BotCompany.de // Java-based operating systems ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T581199cf280badd7-Ma99c1854bafbf0b0c626b23d Delivery options: https://agi.topicbox.com/groups/agi/subscription
