Think if you had a GPT-2 prompt 1 word long only ex. scratching, it could be seen as scratch ing, so let's say we had only scratch, the word to predict next is the one seen before in the data, and the frequent one is more likely chosen. This word scratch can also match related words itch scrap etc and then see what frequent word is next. So the closest related word, and the highest frequent word is chosen.
scratch the x64 scratch it x58 scratch back x38 itch my x76 itch shoulder x50 So my is predicted because itch=scratch a lot and my is seen after itch x76 times. But we aren't done yet. The candidate words/tokens are given an extra score, now, which is relational score to story words 'scratch' (no itch exist) in your input prompt. So we end up picking shoulder say. We may also then translate shoulder to adapt it in ex. stomach if your story was about stomach itch. If we have 2 words in our generated story so far: 'Scratch stomach', we can predict the next word using either word but we want to use both, so we see what nodes/related nodes including positional rearranging exists in the data ex. scratch stomach, itch bladder, bladder itch, etc, and the more distant they are positioned the lesser of course a match it is. So say we got a 'first I go into the lake and stay there, second I cold freeze in the winter', the 'first' votes on then because it could be 1st or etc translated, frequency, relation, but it is far back and there's other words, so it definitely has less vote being 1/10th the weight and fading in energy by this time, yet it does have a strong vote too. Say we find 'stomach itch but it has a word in the middle or one missing ex. 'stomach really itch' or 'stomach', this also has less score but still has some. What if I say 'second, this is that, third, i am this, but i won't say the next because i refuse to repeat myself'. Here I pay attention to ignore the word 'fourth' being said. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T7cbcba9a1ae63532-Me031bc5f185387e5de664d41 Delivery options: https://agi.topicbox.com/groups/agi/subscription