On Mon, Aug 30, 2021, 11:27 PM <[email protected]> wrote: > how does a word predictor work if the spaces are taken out >
The spaces are not taken out. You have contexts that start on word boundaries and fold together upper and lower case by dropping bit 5. If you use a dictionary and remove spaces then the context model parses the input to find the word boundaries using the same encoding that the decompressor uses to restore the spaces. You can also fold related words together. ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T90b7756a48658254-M6c0b3355d6112f1da357abbe Delivery options: https://agi.topicbox.com/groups/agi/subscription
