Here is solid proof the word to predict is heavily influenced by story words no matter where they are in the story. Watch as I force GPT-2 to change the %s as I add words. Notice also if it seen it entail in the story it makes it even more likely. Also, when you talk about trees etc it will also boost up not only trees candidates but grass leafs etc etc. It uses something like Word2vec not only to recognize the sentence (it also focuses on important words so summarize it so it), but also for voting on the prediction candidates, see!
https://www.youtube.com/watch?v=RWd-LBgC9UM&feature=youtu.be ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T1f56a0e2e53cf50a-M8ea57903ee55fe18d18e0a8e Delivery options: https://agi.topicbox.com/groups/agi/subscription
