One way I like to think about patterns is as symmetries that can be observed in some way. No matter what the pattern is, it always follows a consistent symmetry, right?
For instance, take a look at this sequence of numbers: 1, 3, 5, 7, 9. What symmetries can you observe?: - All integers are positive True, but it doesn't show us every symmetry. Here's another one: - We can add two to the current positive integer to get the next positive integer The second symmetry has more dimensions or meaning encoded into it, because we can also reduce it into the first symmetry of "all integers are positive". In the example above, we're "looking for" or "finding" symmetries. From a Computer Science lense, that sounds like a search problem. If it's a search problem, that implies you also have to know what you're looking for before you look for it. One refinement of this idea is that purely reactive learning could turn into proactive learning as you learn what to expect. In language, you learn that *big* often stands before *book*, and that words which in other ways are like *big* often stand before words that are otherwise like *book*, so you generalise to adjectives standing before nouns; then whenever you hit a word which you think is an adjective, you actively *look for* its noun - a very different learning strategy from purely statistical learning. These expectations gradually turn into a grammar, where you can talk about dependencies and dependency types (e.g. subjects versus objects). I know that sounds like hand-crafting creeping back in through the back door, but all that's hand-crafted is your initial set of pattern types. He tries to find symmetries in a sentence by looking (or searching) for precedence, which to some degree is the same thing we did above in the second symmetry. And the way he describes it as a "gradual turning into grammer" and "dependencies and dependency types" is also kind of analogous to the low/high dimensionality of the symmetries. > But I can't find a better word ... so I'm stuck with "statistical > learning", for now. It really really sucks, since it is a HUGE impediment > to getting the ideas across. > Oh ye minters of catchy phrases, I appeal to you now! What about symmetrical learning or consistency learning? Not very poetic, but the basis of patterns (at least, the way I see them in my crazy head) is consistent symmetries. On Sun, Jul 19, 2020 at 3:11 PM Linas Vepstas <[email protected]> wrote: > Hi Dick, > > Well, yes, exactly! But let me provide some insight. > > On Sun, Jul 19, 2020 at 7:01 AM Richard Hudson <[email protected]> wrote: > >> Another voice from a long-time lurker: patterns are only relevant if they >> can be recognised by humans. Some can, others can't. E.g. I don't think >> humans are good at recognising that cba is the reverse of abc, but I >> imagine a machine might well pick it up. So maybe you need a mixture of >> hand-crafting for the pattern types and uncontrolled searching for those >> patterns. >> > Clearly, you were thinking of "garden-path sentences", which many computer > language systems can parse just fine, but humans stumble over, if not > completely baffled by. "The horse raced past the barn fell". > > What is happening here is an interesting interaction between "metric > distance" and algorithms. Roughly speaking, an algorithm that takes N > steps to find something means that the "something is distance-N away from > the trivial case". There's also an idea of "relative distance": for > example, two complex sentences that differ by a single-word substitution > can be judged as being close to each other, instead of far apart (they are > not 2N-apart). > > Different algorithms have different embodiments of "distance". The abc- > vs. cba example is famous from computer-science: It is fundamentally > impossible (it's a theorem) for a finite automaton (aka regular-expression > parser) to recognize reversed strings. Reversal is often used as the > textbook example for why push-down automata (aka "context-free languages") > are needed, and how/why they differ from finite state machines. > > The success of deep-learning and neural nets is *entirely* due to these > having a very dramatically different distance metric. Vastly different -- > famously, reasoning by analogy: "King is to Queen as man is to XXX" and > neural nets can solve this problem instantly (I think this is an example > from one of the foundational papers) whereas attacking this problem by > building traditional parsing and synonym-learning takes vast amounts of > CPU, complexity, is error prone, opaque, and has poor results in general. > > This is part of what I think about, what I mean, when I say "pattern". > Perhaps the "blind men and the elephant" story: if you touch the elephant, > you get a rough surface. If you smell it, it .. smells like (whatever > elephants smell like). If you have extremely bad blurry vision, you see a > ball of grey that moves on a background. What does grey have to do with > rough texture? These different sense-perceptions are like different > algorithms. The "patterns" they are good at are wildly different. There's > no one algorithm to rule them all. > > I think there is a deep mathematical basis for all this, but I can't > sketch it here. I draw insight from operator algebras, invariant measures, > spectra of transfer operators, sofic shifts and the PBW theorem for > representation theory and tensor algebras. These, in a certain sense, > provide tools for thinking about algorithms and the distances between the > things that the algorithm can visit/discover/explore. The simplest > examples include geometric state machines over cantor sets, which work with > certain very simple fractals (ergodic systems, Bernoulli shifts) as their > "natural domain". So -- fractals look very complicated, until you know > what their "secret" is, and then many of them become "simple". Even though > the act of "recognizing" them ("which fractal is this?") is hard. > > Roughly speaking, the relationship between a "grammar" and the language it > generates (the set of all syntactically-valid sentences) is the same > relationship as between a fractal-generator, and the fractal it produces. > Learning a grammar is like deducing the fractal generator, given samples of > fractals. Neural nets just provide one particular viewpoint into solving > this problem. > > Some of what I am saying can be made very concrete in a rather very pretty > way. Thiryt+ years ago, Przemyslaw Prusinkiewicz started out with very > simple grammars generating very simple fractals, and over the decades, he > has ended up with this: http://algorithmicbotany.org/papers/ Enjoy the > pictures, but .. > > But ... if you are at all serious about AGI, or about "solving biology", > one of the most important things you can do is to start reading his papers > from 1985, onwards, through the late 1980's, early 1990's .. late 1990's > ... the 2000's ... and give them deep attention and thought. The deep > secrets of logic, reasoning, natural language, common sense and human > understanding are buried in there, even though it is superficially labelled > "botany". > > One refinement of this idea is that purely reactive learning could turn >> into proactive learning as you learn what to expect. In language, you learn >> that *big* often stands before *book*, and that words which in other >> ways are like *big* often stand before words that are otherwise like >> *book*, so you generalise to adjectives standing before nouns; then >> whenever you hit a word which you think is an adjective, you actively *look >> for* its noun - a very different learning strategy from purely >> statistical learning. These expectations gradually turn into a grammar, >> where you can talk about dependencies and dependency types (e.g. subjects >> versus objects). I know that sounds like hand-crafting creeping back in >> through the back door, but all that's hand-crafted is your initial set of >> pattern types. >> > Well, I think this is *exactly* what I'm trying to do with > https://github.com/opencog/learn and so perhaps the use of the word > "statistical" is misleading. There's a problem with the word "statistics", > and that is, that, for most people, it's about those stunningly dull and > boring books you may have been forced to read, talking about expectation > values and standard deviations and hypothesis testing and p>0.05 confidence > levels. The reality of statistics is very very different. > > But I can't find a better word .. for example, quantum mechanics is > 99.999% "statistics", but if you use the word "quantum", people think of > something totally different. If you say "fractals", its 99.999% the > statistical nature of the cantor set, but most people don't know that, and > think it's something else. If you look at the distribution of chaotic > orbits on riemann surfaces and the Artin zeta: well, what the hecke-algebra > -- that's 99.9999% statistics again, but when you say statistics, most > people don't think "oh, riemann surfaces, modular forms, I see what you > mean". So I'm at a loss for the correct word to use for the kind of > statistical learning I am talking about. Calling it "quantum fractal > modular surface learning" is only something a brazen marketing agency could > come up with, and so I'm stuck with "statistical learning", for now. It > really really sucks, since it is a HUGE impediment to getting the ideas > across. > > Oh ye minters of catchy phrases, I appeal to you now! > > -- Linas > >> Best wishes for your thinking. Dick >> On 19/07/2020 01:26, Linas Vepstas wrote: >> >> The word "training" is problematic. If you mean "memorize an association >> list of pairs" (e.g. faces+text-string) well, technically that is >> "training" in the AI jargon file, but it's of little utility for AGI. >> >> The word "pattern" is problematic. Exactly what a "pattern" is, is ... >> tricky. Much (most? almost all?) of my effort is about trying to define >> "what is a pattern, anyway". I'm not sure what you had in mind, when you >> used that word. (Its a tricky word. Everyone obviously knows what it >> means, but how to turn it into an algorithmically graspable "thing"?) >> >> --linas >> >> On Sat, Jul 18, 2020 at 6:44 PM Dave Xanatos <[email protected]> wrote: >> >>> "If you can't spot the pattern, you've not accomplished anything." >>> >>> >>> >>> Every significant – and truly useful - advance I've made on my own >>> language apprehension code has been based on recognizing a pattern, and >>> coding for it. I fully agree. >>> >>> >>> >>> Can a neural network be trained on patterns instead of things? >>> >>> >>> >>> Can code designed to recognize – for example, faces (like eigenfaces) – >>> be trained to instead recognize blocks of data that look the same, despite >>> perhaps being in vastly dissimilar fields? >>> >>> >>> >>> Apologies if I'm intruding, or seem to be "out of my lane"… a popular >>> buzzword these days. >>> >>> >>> >>> Dave – LONG time lurker… >>> >>> >>> >>> >>> >>> >>> >>> *From:* [email protected] <[email protected]> *On Behalf >>> Of *Linas Vepstas >>> *Sent:* Saturday, July 18, 2020 6:54 PM >>> *To:* link-grammar <[email protected]>; opencog < >>> [email protected]> >>> *Subject:* [opencog-dev] Re: [Link Grammar] Sutton's bitter lesson >>> >>> >>> >>> Well yes. What's truly remarkable is how frequently that lesson has to >>> be re-learned. There are vast swaths of the AI industry that still have >>> not learned it, and are deluding themselves into thinking that they've made >>> bold progress, when they've gotten nowhere at all, and seem blithely >>> unaware that they are repeating the same mistake... again. >>> >>> >>> >>> I refer, of course, to the deep-learning true-believers. They have made >>> the fundamental mistake of thinking that their various network designs >>> provide an adequate representation of reality. How little do they seem to >>> realize that all that code, running hand-tuned on some GPU is just, and I >>> quote Sutton, here: "leveraged human understanding of the special >>> structure of chess". Except, cross out "chess" and replace with >>> "dimensional reduction" or "weight vector" or whatever buzzword-bingo is >>> popular in the deep-learning field these days. >>> >>> >>> >>> I'm back again to insisting that "patterns matter". If you can't spot >>> the pattern, you've not accomplished anything. Neural nets can't spot >>> patterns. They're certainly interesting for various reasons, but, as an AGI >>> technology, they are every bit a dead-end as the hand-crafted English >>> link-grammar dictionary. >>> >>> >>> >>> This is one reason I'm sort of plinking away, working on unfashionable >>> things. I'm thinking simply that they are more generic. and more powerful. >>> But perhaps the problem is recursive: perhaps I'm just "leveraging my human >>> understanding of the special structure of patterns", and will hit a wall >>> someday. For now, it seems that my wall is more distant. If only I could >>> convince others ... >>> >>> >>> >>> --linas >>> >>> >>> >>> >>> >>> On Sat, Jul 18, 2020 at 5:14 PM Paul McQuesten <[email protected]> >>> wrote: >>> >>> Linas, >>> >>> >>> >>> I think this reinforces your view of learning from data, instead of >>> adding more human-curated rules: >>> >>> http://incompleteideas.net/IncIdeas/BitterLesson.html >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "link-grammar" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/link-grammar/464d1f92-00b7-4780-870a-2156229b4567o%40googlegroups.com >>> <https://groups.google.com/d/msgid/link-grammar/464d1f92-00b7-4780-870a-2156229b4567o%40googlegroups.com?utm_medium=email&utm_source=footer> >>> . >>> >>> >>> >>> -- >>> >>> Verbogeny is one of the pleasurettes of a creatific thinkerizer. >>> --Peter da Silva >>> >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "opencog" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/opencog/CAHrUA36x8QBXGUg4f9BMw5StdhRu1WFjFr_9ySo_vZesMeZrTA%40mail.gmail.com >>> <https://groups.google.com/d/msgid/opencog/CAHrUA36x8QBXGUg4f9BMw5StdhRu1WFjFr_9ySo_vZesMeZrTA%40mail.gmail.com?utm_medium=email&utm_source=footer> >>> . >>> -- >>> You received this message because you are subscribed to the Google >>> Groups "opencog" group. >>> To unsubscribe from this group and stop receiving emails from it, send >>> an email to [email protected]. >>> To view this discussion on the web visit >>> https://groups.google.com/d/msgid/opencog/002701d65d5d%244fdc07d0%24ef941770%24%40xanatos.com >>> <https://groups.google.com/d/msgid/opencog/002701d65d5d%244fdc07d0%24ef941770%24%40xanatos.com?utm_medium=email&utm_source=footer> >>> . >>> >> >> >> -- >> Verbogeny is one of the pleasurettes of a creatific thinkerizer. >> --Peter da Silva >> >> -- >> You received this message because you are subscribed to the Google Groups >> "link-grammar" group. >> To unsubscribe from this group and stop receiving emails from it, send an >> email to [email protected]. >> To view this discussion on the web visit >> https://groups.google.com/d/msgid/link-grammar/CAHrUA368NNEmjBsg_09%3DG7yJOdT2Ur%3DBMYvEZPFh2k_HiWNx7w%40mail.gmail.com >> <https://groups.google.com/d/msgid/link-grammar/CAHrUA368NNEmjBsg_09%3DG7yJOdT2Ur%3DBMYvEZPFh2k_HiWNx7w%40mail.gmail.com?utm_medium=email&utm_source=footer> >> . >> >> -- >> Richard Hudson (dickhudson.com) >> >> >> >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> >> Virus-free. >> www.avg.com >> <http://www.avg.com/email-signature?utm_medium=email&utm_source=link&utm_campaign=sig-email&utm_content=emailclient> >> <#m_8122679212014709731_m_-1737756549167139161_DAB4FAD8-2DD7-40BB-A1B8-4E2AA1F9FDF2> >> > > > -- > Verbogeny is one of the pleasurettes of a creatific thinkerizer. > --Peter da Silva > > -- > You received this message because you are subscribed to the Google Groups > "opencog" group. > To unsubscribe from this group and stop receiving emails from it, send an > email to [email protected]. > To view this discussion on the web visit > https://groups.google.com/d/msgid/opencog/CAHrUA3756umETANn42ymt6eQDy6SV6Utu98gSO7%3DH%2BWqBOEiwg%40mail.gmail.com > <https://groups.google.com/d/msgid/opencog/CAHrUA3756umETANn42ymt6eQDy6SV6Utu98gSO7%3DH%2BWqBOEiwg%40mail.gmail.com?utm_medium=email&utm_source=footer> > . > -- You received this message because you are subscribed to the Google Groups "opencog" group. To unsubscribe from this group and stop receiving emails from it, send an email to [email protected]. To view this discussion on the web visit https://groups.google.com/d/msgid/opencog/CAPPXERrRNwM7LcwqfoeD9QTAiBTWVoCHuakk2XCYvBc1BScxSg%40mail.gmail.com.
