Dictionary preprocessing helps by reducing the input size, which reduces memory usage, and by effectively increasing the context length without increasing the number of contexts that need to be mixed.
On Sat, Aug 28, 2021, 6:23 AM <[email protected]> wrote: > I shuffled the 80, 3840, and the rest of the words in english.dic (keeping > them in 3 groups of 1/2/3 bytes) to simply rid the related words grouping, > and got some interesting scores (decompression successful, exact match). So > the grouping seems to help some amount but without it the preprocessor > still hands a big ton of help. I'm wondering if this is due to it being > like Byte Pair Encoding making it focus on/ predict chunks instead of > letters? And/or is it the smaller codes given to more common features? BUT > how can that be so? Isn't the PPM algorithm supposed to do that by making > frequent features predicted more heavily? Why does making the into a single > byte help compression when my AI already knows t/th/the/he/e are very > frequent. > > bytes fed in > 100,000 > 1,000,000 > > without preprocessor > 28,081 > 240,741 > > with [shuffled] preprocessor > 24,684 > 221,312 > > with preprocessor > 24,202 > 217,575 > > *Artificial General Intelligence List <https://agi.topicbox.com/latest>* > / AGI / see discussions <https://agi.topicbox.com/groups/agi> + > participants <https://agi.topicbox.com/groups/agi/members> + > delivery options <https://agi.topicbox.com/groups/agi/subscription> > Permalink > <https://agi.topicbox.com/groups/agi/T192296c5c5a27230-M542c1f7fa17ad06f48a48014> > ------------------------------------------ Artificial General Intelligence List: AGI Permalink: https://agi.topicbox.com/groups/agi/T192296c5c5a27230-M228ef4c0cafdb65d961f9fa6 Delivery options: https://agi.topicbox.com/groups/agi/subscription
